Why are the remote bytes shown different from the local bytes?

Need help with FileZilla Client? Something does not work as expected? In this forum you may find an answer.

Moderator: Project members

Post Reply
Message
Author
douglerner
425 Can't open data connection
Posts: 43
Joined: 2017-09-30 12:39
First name: Doug
Last name: Lerner

Why are the remote bytes shown different from the local bytes?

#1 Post by douglerner » 2019-10-24 23:55

With 15 files from just a few hundred bytes in size to a few hundred KB in size, all text files, the local bytes shown match exactly with the remote bytes after upload.

But this morning I was testing a .txt file that is 5,833,434 bytes. When I upload it with FileZilla it shows on the remote server as 5,935,888 bytes.

If I do the same test with Forklift it shows the same 5,833,424 bytes on the remote server.

I downloaded and renamed the FileZilla-uploaded file and after downloading it shows the same as the original 5,833,434 bytes on my desktop, and if I compare the two files they are identical.

Weird.

Why does the file uploaded via FileZilla show a different number of bytes on the remote server even though they apparently are the same file?

Thanks.

douglerner
425 Can't open data connection
Posts: 43
Joined: 2017-09-30 12:39
First name: Doug
Last name: Lerner

Re: Why are the remote bytes shown different from the local bytes?

#2 Post by douglerner » 2019-10-25 02:16

I tested with Transmit too. It also shows 5.8 MB at the remote server after upload. Only FileZilla seems to show 5.9 MB at the remote server after upload for some reason.

If I open FileZilla and look at the remote directory after uploading from Forklift or Transmit, it does show the right number of bytes. It's only when FileZilla itself uploads the file that it shows the wrong number of bytes on the remote side.

Any idea why? Some character conversion or something going on with a file of .txt extension?

douglerner
425 Can't open data connection
Posts: 43
Joined: 2017-09-30 12:39
First name: Doug
Last name: Lerner

Re: Why are the remote bytes shown different from the local bytes?

#3 Post by douglerner » 2019-10-25 02:31

Just some more analysis of this strange behavior.

Going in via the shell to see what's there after uploads I see this:

-rw-r--r-- 1 root root 5833434 Oct 25 02:25 schoolList.txt
-rw-r--r-- 1 root root 5935888 Oct 25 02:22 schoolList2.txt

I renamed the FZ uploaded file to schoolList2.txt. It has more bytes.

A wc comparison shows:

102454 873047 5833434 schoolList.txt
102454 873047 5935888 schoolList2.txt

So both files have the same number of words, same number of lines, but there are just more bytes in the FZ uploaded file for some reason.

Yet... if I then download schoolList2.txt back to my Mac, the number of bytes on my Mac again match that of schoolList.txt.

I'd love to understand what is going on.

Thanks,

doug

douglerner
425 Can't open data connection
Posts: 43
Joined: 2017-09-30 12:39
First name: Doug
Last name: Lerner

Re: Why are the remote bytes shown different from the local bytes?

#4 Post by douglerner » 2019-10-25 02:39

I've narrowed down the cause. It has to do with the way FZ handles files with a .txt extension. As a test, I changed the extension from .txt to .auto and the bytes match up perfectly when uploaded from FZ:

-rw-r--r-- 1 root root 5833434 Oct 25 02:35 schoolList.auto <- from FZ
-rw-r--r-- 1 root root 5833434 Oct 25 02:25 schoolList.txt <- from ForkLift
-rw-r--r-- 1 root root 5935888 Oct 25 02:22 schoolList2.txt <- from FZ, for some reason it ends up with more bytes in the file if it is a .txt file.

So know that we know it is due to FZ doing something to the .txt files is there a way of avoiding that behavior and having FZ treat .txt files like other files?

Thanks,

doug

douglerner
425 Can't open data connection
Posts: 43
Joined: 2017-09-30 12:39
First name: Doug
Last name: Lerner

Re: Why are the remote bytes shown different from the local bytes?

#5 Post by douglerner » 2019-10-25 03:25

Brainstorm. I bet I know what FZ is doing. It's adjusting the character used for carriage-return-line-feed in both directions because it knows it's clearly a text file from the .txt extension.

Looking at the wc results again:

102454 873047 5833434 schoolList.txt
102454 873047 5935888 schoolList2.txt


5935888 - 5833434 = 102455, exactly the number of lines.

So it must be turning Mac \n into \r\n characters when uploading and doing the reverse on the way back.

I haven't tested, but it probably will do the same thing for html files as well.

The .auto files I was uploading were not a known extension, so FZ just uploading them as is.

Is there an option to not adjust line feeds?

Thanks,

doug

douglerner
425 Can't open data connection
Posts: 43
Joined: 2017-09-30 12:39
First name: Doug
Last name: Lerner

Re: Why are the remote bytes shown different from the local bytes?

#6 Post by douglerner » 2019-10-25 04:06

And solved. I just changed the transfer type from auto to binary and it works like other clients now.

I think all modern editors can deal with the different line endings anyway.

Whew.

doug

User avatar
boco
Contributor
Posts: 26910
Joined: 2006-05-01 03:28
Location: Germany

Re: Why are the remote bytes shown different from the local bytes?

#7 Post by boco » 2019-10-25 04:38

Think again. There are still many software programs that cannot deal with foreign line-ending styles correctly. Some Linux distros use very legacy packages with only backported fixes.

What FileZilla does is standards-compliant. A slight change in file size is correct and expected if the server uses a different OS as the client. Linux and Mac use one byte for line endings while Windows and FTP, internally use two.

See Data Type.
No support requests over PM! You will NOT get any reply!!!
FTP connection problems? Please read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
FileZilla Pro support: https://customerforum.fileZilla-project.org

douglerner
425 Can't open data connection
Posts: 43
Joined: 2017-09-30 12:39
First name: Doug
Last name: Lerner

Re: Why are the remote bytes shown different from the local bytes?

#8 Post by douglerner » 2019-10-25 04:41

Since Linux and Mac both use one char line endings, and since the remote server is Linux, and my computer is Mac, I wonder why the number of bytes changed then...

User avatar
boco
Contributor
Posts: 26910
Joined: 2006-05-01 03:28
Location: Germany

Re: Why are the remote bytes shown different from the local bytes?

#9 Post by boco » 2019-10-25 04:44

In that case, the files would be changing file size only if they weren't in native Linux format, before. E. g. if you upload text files created by Windows (CRLF) from a Linux system, the size will change, nonetheless.
No support requests over PM! You will NOT get any reply!!!
FTP connection problems? Please read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
FileZilla Pro support: https://customerforum.fileZilla-project.org

douglerner
425 Can't open data connection
Posts: 43
Joined: 2017-09-30 12:39
First name: Doug
Last name: Lerner

Re: Why are the remote bytes shown different from the local bytes?

#10 Post by douglerner » 2019-10-25 04:52

If the original file was created in Windows (CRLF) and uploaded to Linux wouldn't you expect the file size to get smaller then after the conversion? It got larger, by one byte per line.

User avatar
boco
Contributor
Posts: 26910
Joined: 2006-05-01 03:28
Location: Germany

Re: Why are the remote bytes shown different from the local bytes?

#11 Post by boco » 2019-10-25 05:03

Please note that it is up to the server. If the FTP server is configured to either use CRLF or not doing local line ending conversion at all, all files uploaded to that server stay in the format used internally by FTP, which is also CRLF.

In other words, ASCII line ending conversion at upload is a two-step process. First, the client converts all text files to use CRLF. Then, the files are transferred. Finally, the server converts them back into whatever native line endings the target OS has. If the server has a bug or is not configured to perform that step, all files stay in CRLF format.

For such buggy or mis-configured servers, ensure the text files are in proper line ending scheme before upload, and then, upload using Image type (Binary).
No support requests over PM! You will NOT get any reply!!!
FTP connection problems? Please read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
FileZilla Pro support: https://customerforum.fileZilla-project.org

douglerner
425 Can't open data connection
Posts: 43
Joined: 2017-09-30 12:39
First name: Doug
Last name: Lerner

Re: Why are the remote bytes shown different from the local bytes?

#12 Post by douglerner » 2019-10-25 05:11

It's interesting that the other FTP clients I've tried: YummyFTP (alas, no longer works under Catalina), Forklift, and Transmit all leave the line endings as they were apparently. This is the first client I've ever seen where the file sizes change.

I did try some tests after uploading as binary. For example, I can edit the uploaded file in the shell on the Linux side with vi and it looks fine. So I guess that editor in Linux must be able to deal with different line endings as well.

I'll give this some thought. Thanks!

Post Reply