Maximum inbound file size?

[neriya]

I have a recurring problem and I'm hoping someone can point me in the right direction. I use Filezilla, both client and server, for many things, but one of the important ones is as a FTP server target where backups of some of our clients are sent. These take the form of large .spf files - ranging in size from only 20 GB to over 1 TB, just depending on the particular system. Most of the time, if the .spf file is too large, we send it via Fedex because it would take too long to send over the internet. However, some clients have *great* internet connections and could upload a large file. However, there appears to be some kind of limiting factor in place. This is my log:

(005016)12/19/2016 22:57:40 PM - user (155.303.277.129)> SIZE D_VOL-b003.spf.tmp
(005016)12/19/2016 22:57:40 PM - user (155.303.277.129)> 213 526404353719
(005016)12/19/2016 22:57:40 PM - user (155.303.277.129)> PASV
(005016)12/19/2016 22:57:40 PM - user (155.303.277.129)> 227 Entering Passive Mode (10,0,0,174,120,149)
(005016)12/19/2016 22:57:40 PM - user (155.303.277.129)> REST 526404353719
(005016)12/19/2016 22:57:40 PM - user (155.303.277.129)> 350 Rest supported. Restarting at 526404353719
(005016)12/19/2016 22:57:40 PM - user (155.303.277.129)> STOR D_VOL-b003.spf.tmp
(005016)12/19/2016 22:57:40 PM - user (155.303.277.129)> 150 Opening data channel for file upload to server of "/server/D_VOL-b003.spf.tmp", restarting at offset 526404353719
(005016)12/19/2016 22:57:40 PM - user (155.303.277.129)> 550 can't access file.

I've obfuscated the IP address and username, but everything else is untouched.

The file in question - D_VOL-b003.spf.tmp - is nearly 1 TB in size. The system was able to upload approximately 490 GB of it before suddenly being cut off. All further attempts at resuming the upload end like this. It doesn't seem to matter which FTP client I use to make the upload, they all cannot proceed any further.

I should post my system stats:
Software: FileZilla Server 0.9.59 beta
Platform: Windows Server 2012 R2 x64
Specs: 16-core, 32 GB RAM, volume where the file is being stored has 17.7 TB free space (of 31.9 TB total)


Any thoughts?

[botg]

There is no size limitation in FileZilla Server. It appears the file is still in use by some other program, that's why attempts to resume fail.

[neriya]

I can confirm the file is not in use by any other program on either side of the transfer. I performed this confirmation by running Process Explorer on the systems on both ends of the connection and looking for open file handles for the file in question and found none.

Based on the log, which side do you think is attempting to hold a lock on the file? In this particular example, the uploading client is not FileZilla, but instead the backup software's own built-in FTP file transfer utility.

Additionally, this issue does not occur for any file transfer under 400 GB. I haven't extensively tested to see if it always stops at this *exact* point - we don't have many clients capable of either generating or uploading a 500+ GB file - but it *has* happened more than once with more than one client. Transfer of 100-300 GB files is relatively common by comparison, and always completes flawlessly.

I went ahead and connected to the sending backup system and attempted a transfer by way of FileZilla FTP client, which also failed. This is the log output:

Response: 227 Entering Passive Mode (234,12,34,567,117,120)
Command: REST 526404353719
Response: 350 Rest supported. Restarting at 526404353719
Command: STOR D_VOL-b003.spf.tmp
Response: 150 Opening data channel for file upload to server of "/server1/D_VOL-b003.spf.tmp", restarting at offset 526404353719
Error: Could not write to transfer socket: ECONNRESET - Connection reset by peer
Response: 550 can't access file.
Error: File transfer failed after transferring 524,288 bytes in 1 second

I'm not saying the problem is 100% for sure coming from the FileZilla Server software - I'm just asking for help. If there's something else generating a limit I'd love to hear about it. This Windows Server 2012 R2 installation is only a couple weeks old and has essentially zero other software installed. This system's entire job in life is to be a giant FTP-enabled fileserver and just hold copies of backups. There's not even any antivirus in play. And I know for 100% certainty that if I ship out a USB hard drive, I can copy this file to it from the generating server, ship it across the country, and copy the file into the same spot I'm trying to FTP it and it'll work just fine.

[botg]

As the message comes from the server it's a server-side problem.

Is there enough free disk space on the server?

As for the offset 526404353719, it's a totally unremarkable number, nothing special about it.

Have you tried checking the server's harddisk for errors? Do an exhaustive check with chkdsk /R


You could also use Sysinternals Process Monitor to find out which Windows API function fails when trying to upload.

[neriya]

There's 17 TB of free disk space, so plenty free.

The server's hard disk is a RAID 6 array of 24 3TB drives. I have not run a chkdsk /R recently; it'd take quite a while I would assume.

I will try Process Monitor and see what it says, thanks!

[boco]

Could you check if the transfers always stop after a specific time (as opposed to size)? Unfortunately, there's poorly designed network equipment that always terminates connections after a fixed time if it 'thinks' they are being idle. What adds to the problem is that the endpoints often aren't notified about the termination and thus not even aware it has been dropped.

The lowest allowed time span for such a connection (as allowed by the TCP/IP specs) is 2 hours + 4 minutes (socket 'cool-down' period before it can be re-used). Peers can send TCP keep-alive messages, but only every 2 hours. That means, with a properly configured router timeout, the connection can be kept open. If the router timeout is below 2 hours, it cannot.

[neriya]

There is definitely no time correlation. Additionally, even if that was happening, I would think the resumes would be able to start.

[botg]

The error message pretty much rules out any time related issue.

[boco]

Probably the virus scanner immediately starts to scan the partial file as soon as the handle is released (file no more locked). The client attempts to resume while the scanner still checks the file (file locked). Common problem with background scanners.

[neriya]

As has already been mentioned, there is no antivirus software installed. And again, even if there was antivirus software, eventually the scan would finish and subsequent resume attempts would function.

I am sick, and have not yet had the health / time to run Process Monitor and see if I can observe the failure live. When I do, I'll update this thread.

[mx957rl]

Bump on this thread.

Any Resolution? I am running into the same issue.

Large files that may have been restarted a few times eventually get the Write Failure.

[neriya]

I never did find a resolution, no. I eventually transferred the file via external hard drive, obviating the need to transfer it via FTP. I presume I would run into the same issue again were I to attempt to transfer a similarly large file in the future, but fortunately 500+ GB files are few and far between for me.