Are my files too big?

[Ziggy32]

I have been using Filezilla for a million years and have never had this problem. I also have a server that is running Filezilla Server. It has to have something to do with the size of the files I am trying to transfer.

At right about the 3GB mark it disconnects with ECONNABORTED. At that point it will reconnect but I can't resume. It gives me a 550 Permission Denied. It has to be the files sizes. It has worked flawlessly for years but I have never tried to transfer anything this big and as mentioned it disconnects at about the 3GB mark every time.

Any ideas? I am running the latest client and server> Were might the problem be?

[botg]

There's no artificial size limitation in FileZilla. The technical limit is 2^64 bytes, that's about 5 billion times more than what you are trying to transfer.


Are you sure it's the size of the file and not the duration of the transfer? Unfortunately many firewalls and NAT routers are utterly broken and drop connections that are idle for less than 2 hours and 4 minutes, the minimum allowed time below which a firewall or NAT router must not drop connections, as per RFC 5382.


Ordinarily this wouldn't be too bad, however it seems that your server does not allow resuming transfers, so recovery isn't possible.

[Ziggy32]

Well.... I just tried the transfer again. It stopped at the exact same place. Right at 3GB. I did a different file two weeks ago and I would swore it stopped at the same place to the byte. Since it stops at the exact same place every time and it is coming from the same remote the duration would have to be the same too but here is the interesting thing...

From looking at the Filezilla Server the duration was two hours almost to the second but it did not disconnect first. The log on the client is the same:

(000003)10/21/2015 14:11:17 PM - jimbecher@microage.local (75.150.230.150)> PASV
(000003)10/21/2015 14:11:17 PM - jimbecher@microage.local (75.150.230.150)> 227 Entering Passive Mode (192,168,1,2,220,206)
(000003)10/21/2015 14:11:18 PM - jimbecher@microage.local (75.150.230.150)> REST 3101557604
(000003)10/21/2015 14:11:18 PM - jimbecher@microage.local (75.150.230.150)> 350 Rest supported. Restarting at 3101557604
(000003)10/21/2015 14:11:18 PM - jimbecher@microage.local (75.150.230.150)> STOR BryanSmedbergOld.pst
(000003)10/21/2015 14:11:18 PM - jimbecher@microage.local (75.150.230.150)> 550 Permission denied
(000003)10/21/2015 14:12:18 PM - jimbecher@microage.local (75.150.230.150)> disconnected.

Another thing I was curious about... two minutes in to the transfer both the client and server show:
(000004)10/21/2015 14:58:37 PM - jimbecher@microage.local (75.150.230.150)> 421 Connection timed out.
(000004)10/21/2015 14:58:37 PM - jimbecher@microage.local (75.150.230.150)> disconnected.

but continue as if nothing has happened.....

[Ziggy32]

I fumbled around long enough I believe I found the answer. Append permissions were not checked on the Filezilla Server. I am going to have to assume that the 3GB mark is about where it actually writes (appends) the cache to the actual file. I checked the Append permissions and it just crossed the 4GB mark.

[botg]

I am going to have to assume that the 3GB mark is about where it actually writes (appends) the cache to the actual file.

That's bullshit. There is no cache in FileZilla Server. This issue has absolutely nothing to do with filesize.

[Ziggy32]

You are totally incorrect. I sat there and watched the folder that it was uploading to. It created the file at the start of the upload. The file size was zero until the 3GB mark (two hours in to the upload). At that the 3GB mark it finally "appended" the 3GB to the file and the file size then became 3GB then the whole process started over again to get it to 6GB then to 9GB until it was finished uploading.

The "Append" had to be checked because that is essentially all it did. It created the file at 0 bytes and did nothing but append to it about every 3GB until the upload finished. It might not be Filezilla Server caching but something sure is.

FYI Filezilla Server is running on a Windows 2012 Server. Now that the Append permission in Filezilla Server is checked the uploads continue past 3GB and completes.

[botg]

Windows caches displayed file sizes and only updates the display if a file handle gets closed. However the file is in fact growing all the time while it is being written to, you just don't see it.

Again, are you sure it is size related and not time related? Try setting a speedlimit to half your usual speed. Then it should fail at roughly half the amount transferred.

You are totally incorrect.

If you say so. I suppose you have more experience than me in diagnosing FTP issues.

[Ziggy32]

All I can tell you is what I saw and what fixed the problem. The uploads would work for about two hours, roughly 3GB, and then come back with a permissions denied error and disconnect.

The fix was to turn on append rights on the Filezilla server. For the first time the upload has gone well past the 3GB mark and has been uploading for 16 hours straight. Don't know what to tell you.

[boco]

What you see is a symptom, not the actual cause. A security or similar software (or driver problem), especially if 32bit, seemingly causes the file handle to close, breaking the transfer. As a result, you see the file size updated.

I tested that yesterday by transferring a 466GiB file. Was already beyond 100GiB without a single problem (no APPEND/REST allowed for that test). Then I aborted it (took too long). But it did not abort by itself.

[Ziggy32]

Yes. At the 3GB mark I get a permissions denied error which closes the handle. But... the only thing I have changed is the Append privilege on the Filezilla Server. I have uploaded hundreds of files with no problems and it is now apparent it is because those files were under a certain size which did not require the append privilege. Simply the write privilege.

At some point (as near as I can tell about 3GB) the append privilege is required. At a certain size (3GB) it goes in to "append mode" where just the write privilege isn't enough. You need write and append.

[botg]

Yes. At the 3GB mark I get a permissions denied error which closes the handle

You got it backwards. Your connection fails, as result the handle is closed. And when FileZilla tries to resume, then the server says permission denied.


Gotta ask a third-time: Why are you still so sure it is the size of the file and not the time it takes to transfer?

[Ziggy32]

Because all I did. The only thing I did. With out a doubt the only thing I changed was granting the append privilege. In watching the client several times it use to stop at the same byte count every time (which granted would also be the same time duration) but after granting the append permission.... it has now been uploading 15 hours and 13 GB and still cooking.

Once this upload is done I could uncheck the append privilege if you would like. Happy to try it. BTW love Filezilla.

[xeon]

At that the 3GB mark it finally "appended" the 3GB to the file and the file size then became 3GB then the whole process started over again to get it to 6GB then to 9GB until it was finished uploading.

According to your post quoted above, it also stops with the append permission granted. The only difference is now you've allowed it to resume when it gets cut off, the problem itself still remains, it just no longer has the side effect of leaving incomplete transfers around.

[Ziggy32]

If your comment about Windows not updating the file size until the handle is closed I would agree. When I granted the append permission I could watch the upload folder and see the file size increase in increments of about 3GB every 2 hours until the upload was complete.

In looking back through the server logs you are correct. It was a continuous series of disconnects for a day and a half. The shortest time between disconnects being a half an hour the longest time between disconnects being an hour and a half. So the append permission really just made the auto resume work which I guess would make sense because at a resume it truly has to append to a file that already exists.

So where does that leave us? You need the append permission to auto resume?

[botg]

Yes. When resuming you are essentially appending data to the end of the file. That requires append permission.