My files are getting corrupted after transfer from server

[arian_dd]

Hi everyone,

I have been using FileZilla close to a year now. I am experiencing problems in transferring files from server to my local computer. I am transferring massive amounts of text files (GBs) to my local computer from a computer center to my local machine and I am performing some post-processing on the files on my local machine. Strangely enough some of the files contain special characters that are not supposed to be there. The copy on the server does not have this problem. Simple SCP command transfers the files without a problem but much more slowly obviously.
I was wondering if anyone has had this problem and any insights on what may cause this problem. Does compressing the files before the transfer help? Are there better solutions?

BTW: I have checked all my codes and none of them have write access to the file.

Thanks

[audiopro]

What type of files are they?
What code is being added?
Are you transferring Binary or Ascii?

[arian_dd]

I am transferring text files (ASCII) files. I usually copy the entire folder and they transfer through the queue. Number of simultaneous connections is left as default==3.
Post processing code that I run on these files is done after the transfer and does not interfere with the transfer process.

[arian_dd]

The "Default Transfer Type" under settings is set to "Auto" though.

[audiopro]

What code is being added?

[arian_dd]

Would you elaborate on your question? You mean what characters are being added? If so there random characters such as *,#,< , ^ etc...

[audiopro]

Are they always the same characters?
Are they added at a particular place?
When repeatedly transferring the same file, are the same chars inserted into the same place?
Do you only get the chars added when transferring large files?
Do you get the same problem with firewall/security software disabled?

As we are unable to replicate this problem for ourselves, you will have to do the experiments to narrow the cause down.

[boco]

Another: FTP or SFTP?

[arian_dd]

@audiopro:
-Characters are not the same in each corrupt file.
-There is no particular pattern for them. They are added anywhere in the file. But there aren't many of them. There are usually 20-30 characters added in each corrupt file.
-When I repeat transferring the individual files the problem does not exist. It is the batch transferring of the files that causes the issue (i think).
-No. There are these weird characters (showing up in a box with 0's and ones') that are representing space in linux.
-I don't have a firewall. I am on ubuntu.

@boco:
Here is what I get from message log when I login: "Response: fzSftp started"
How can I check that?
I

[audiopro]

I had a similar problem quite a while ago with an SFTP application and the problem was eventually tracked down to certain chars being changed during the encoding/decoding process. I managed to isolate a couple of characters which were being changed by creating files containing regular patterns of the full charset and then comparing the transferred files with the originals.
Do you know if chars are being inserted or are some chars being changed.
Sorry if it is long winded but it is far easier tracking these problems down when you have direct access to the files themselves.

[arian_dd]

I use grep to track these "special characters" that are being inserted. Unfortunately they are replacing some of the characters in the file. For instance a float that is supposed to look like 1.23456E-06 changes to 1.23456E-0^/

[botg]

There's no encoding or line-ending conversion performed when using SFTP, files are transferred as-is. Which SFTP server software are you using? (Product and version)

[boco]

Here is what I get from message log when I login: "Response: fzSftp started"
How can I check that?

You just did. fzSftp is SFTP.

FileZilla's SFTP support is based on a third-party component (psftp from the PuTTY suite). Could you try that tool directly? If it produces the same issue the error is not with FileZilla...

[audiopro]

Sorry, my mistake, I thought SFTP was a secure version of FTP which involved some kind of encode/decode.
The transfer program I am referring to did just that but I cannot remember what it was called as it was a long time ago and it was not very good.

[boco]

The secure extension of FTP is called FTPS (Explicit/Implicit), not SFTP.