Please Help Fix: 426 Connection closed; transfer aborted

[mcjathan]

I believe this error is pointing to a malicious router. However, I can't pin-point the problem and fix it. Your help is appreciated.

In the logs below, an FTP client is downloading a file from the FileZilla server. It appears that every 21 or 22 seconds, the connection is closed/aborted, and then the client restarts the download.

Here is the tail end of the FileZilla server log:

(000058)3/25/2013 13:09:10 PM - bestsync (166.70.54.154)> 426 Connection closed; transfer aborted.
(000058)3/25/2013 13:09:10 PM - bestsync (166.70.54.154)> PASV
(000058)3/25/2013 13:09:10 PM - bestsync (166.70.54.154)> 227 Entering Passive Mode (23,30,60,44,195,116)
(000058)3/25/2013 13:09:10 PM - bestsync (166.70.54.154)> REST 27721728
(000058)3/25/2013 13:09:10 PM - bestsync (166.70.54.154)> 350 Rest supported. Restarting at 27721728
(000058)3/25/2013 13:09:10 PM - bestsync (166.70.54.154)> RETR /R-Disk/Mentors/Kudlow, Larry/The Larry Kudlow Show (weekly)/2013_03_16 March_16__2013-1363453308.mp3
(000058)3/25/2013 13:09:10 PM - bestsync (166.70.54.154)> 150 Connection accepted, restarting at offset 27721728
(000058)3/25/2013 13:09:32 PM - bestsync (166.70.54.154)> 426 Connection closed; transfer aborted.
(000058)3/25/2013 13:09:32 PM - bestsync (166.70.54.154)> PASV
(000058)3/25/2013 13:09:32 PM - bestsync (166.70.54.154)> 227 Entering Passive Mode (23,30,60,44,195,117)
(000058)3/25/2013 13:09:32 PM - bestsync (166.70.54.154)> REST 31186944
(000058)3/25/2013 13:09:32 PM - bestsync (166.70.54.154)> 350 Rest supported. Restarting at 31186944
(000058)3/25/2013 13:09:32 PM - bestsync (166.70.54.154)> RETR /R-Disk/Mentors/Kudlow, Larry/The Larry Kudlow Show (weekly)/2013_03_16 March_16__2013-1363453308.mp3
(000058)3/25/2013 13:09:32 PM - bestsync (166.70.54.154)> 150 Connection accepted, restarting at offset 31186944
(000058)3/25/2013 13:09:54 PM - bestsync (166.70.54.154)> 426 Connection closed; transfer aborted.
(000058)3/25/2013 13:09:54 PM - bestsync (166.70.54.154)> PASV
(000058)3/25/2013 13:09:54 PM - bestsync (166.70.54.154)> 227 Entering Passive Mode (23,30,60,44,195,118)
(000058)3/25/2013 13:09:54 PM - bestsync (166.70.54.154)> REST 34652160
(000058)3/25/2013 13:09:54 PM - bestsync (166.70.54.154)> 350 Rest supported. Restarting at 34652160
(000058)3/25/2013 13:09:54 PM - bestsync (166.70.54.154)> RETR /R-Disk/Mentors/Kudlow, Larry/The Larry Kudlow Show (weekly)/2013_03_16 March_16__2013-1363453308.mp3
(000058)3/25/2013 13:09:54 PM - bestsync (166.70.54.154)> 150 Connection accepted, restarting at offset 34652160
(000058)3/25/2013 13:10:15 PM - bestsync (166.70.54.154)> 426 Connection closed; transfer aborted.
(000058)3/25/2013 13:10:16 PM - bestsync (166.70.54.154)> PASV
(000058)3/25/2013 13:10:16 PM - bestsync (166.70.54.154)> 227 Entering Passive Mode (23,30,60,44,195,119)
(000058)3/25/2013 13:10:16 PM - bestsync (166.70.54.154)> REST 38117376
(000058)3/25/2013 13:10:16 PM - bestsync (166.70.54.154)> 350 Rest supported. Restarting at 38117376
(000058)3/25/2013 13:10:16 PM - bestsync (166.70.54.154)> RETR /R-Disk/Mentors/Kudlow, Larry/The Larry Kudlow Show (weekly)/2013_03_16 March_16__2013-1363453308.mp3
(000058)3/25/2013 13:10:16 PM - bestsync (166.70.54.154)> 150 Connection accepted, restarting at offset 38117376
(000058)3/25/2013 13:10:37 PM - bestsync (166.70.54.154)> 226 Transfer OK


Here is the corresponding tail of the FTP client log:

<[2]150 Connection accepted, restarting at offset 24256512
<[2]426 Connection closed; transfer aborted.
>[2]PASV
<[2]227 Entering Passive Mode (23,30,60,44,195,116)
>[2]REST 27721728
<[2]350 Rest supported. Restarting at 27721728
>[2]RETR /R-Disk/Mentors/Kudlow, Larry/The Larry Kudlow Show (weekly)/2013_03_16 March_16__2013-1363453308.mp3
<[2]150 Connection accepted, restarting at offset 27721728
<[2]426 Connection closed; transfer aborted.
>[2]PASV
<[2]227 Entering Passive Mode (23,30,60,44,195,117)
>[2]REST 31186944
<[2]350 Rest supported. Restarting at 31186944
>[2]RETR /R-Disk/Mentors/Kudlow, Larry/The Larry Kudlow Show (weekly)/2013_03_16 March_16__2013-1363453308.mp3
<[2]150 Connection accepted, restarting at offset 31186944
<[2]426 Connection closed; transfer aborted.
>[2]PASV
<[2]227 Entering Passive Mode (23,30,60,44,195,118)
>[2]REST 34652160
<[2]350 Rest supported. Restarting at 34652160
>[2]RETR /R-Disk/Mentors/Kudlow, Larry/The Larry Kudlow Show (weekly)/2013_03_16 March_16__2013-1363453308.mp3
<[2]150 Connection accepted, restarting at offset 34652160
<[2]426 Connection closed; transfer aborted.
>[2]PASV
<[2]227 Entering Passive Mode (23,30,60,44,195,119)
>[2]REST 38117376
<[2]350 Rest supported. Restarting at 38117376
>[2]RETR /R-Disk/Mentors/Kudlow, Larry/The Larry Kudlow Show (weekly)/2013_03_16 March_16__2013-1363453308.mp3
<[2]150 Connection accepted, restarting at offset 38117376
<[2]226 Transfer OK

Finally, I have attached an image of the router's management page that shows various time-out parameters. I don't see any of the time-out parameters that are close to the 21 or 22 seconds time intervals between the "426 Connection closed" errors.

FYI, the router is supposed to be a fairly high-end consumer router. It is the ASUS RT-N66U. The stock firmware has been replaced with a variant of TomatoUSB -- Tomato by Shibby (http://tomato.groov.pl/?page_id=81). This firmware, has a lofty reputation for both stability and features.

Suggestions please!

[boco]

Start with:
-Change both TCP timeout and generic timeout to at least 7440 seconds (2 hours 4 minutes as required by the TCP/IP specs).
-Uncheck the FTP NAT Helper and make sure your server is configured according to Network Configuration (Passive).

[mcjathan]

Start with:
-Change both TCP timeout and generic timeout to at least 7440 seconds (2 hours 4 minutes as required by the TCP/IP specs).
-Uncheck the FTP NAT Helper and make sure your server is configured according to Network Configuration (Passive).

Thanks for your help, boco.

I changed the TCP timeout and generic timeout to 7440 seconds, and I unchecked the FTP NAT helper. I saved the configuration and rebooted the router (see the screenshot below).

I also reviewed the Network Configuration link you provided (I've spent a fair amount of time absorbing that page in the past). For good measure, I used the website https://ftptest.net// on my server and it reported "Congratulations! Your server is working and assorted routers/firewalls have been correctly configured for plain basic FTP."

After these configuration changes and testing, I re-ran my FTP job. The problem still isn't solved. Based on the Filezilla server logs (see below), the disconnect / restart interval seems to be approximately 10 seconds. Strange! Yesterday, the interval was approximately 21 seconds.

Boco, what do you suggest now?

Regards, Jeff

=============================================
FileZilla Server Log Tail:

(000016)3/26/2013 10:30:11 AM - bestsync (166.70.54.154)> 150 Connection accepted, restarting at offset 6778880
(000016)3/26/2013 10:30:21 AM - bestsync (166.70.54.154)> 426 Connection closed; transfer aborted.
(000016)3/26/2013 10:30:22 AM - bestsync (166.70.54.154)> PASV
(000016)3/26/2013 10:30:22 AM - bestsync (166.70.54.154)> 227 Entering Passive Mode (23,30,60,44,195,124)
(000016)3/26/2013 10:30:22 AM - bestsync (166.70.54.154)> REST 8134656
(000016)3/26/2013 10:30:22 AM - bestsync (166.70.54.154)> 350 Rest supported. Restarting at 8134656
(000016)3/26/2013 10:30:22 AM - bestsync (166.70.54.154)> RETR /R-Disk/Mentors/Edelman, Rick/Podcasts/2013_03_16 TruthAboutMoneyRicEdelmanShow031613HourOne.mp3
(000016)3/26/2013 10:30:22 AM - bestsync (166.70.54.154)> 150 Connection accepted, restarting at offset 8134656
(000016)3/26/2013 10:30:33 AM - bestsync (166.70.54.154)> 426 Connection closed; transfer aborted.
(000016)3/26/2013 10:30:34 AM - bestsync (166.70.54.154)> PASV
(000016)3/26/2013 10:30:34 AM - bestsync (166.70.54.154)> 227 Entering Passive Mode (23,30,60,44,195,125)
(000016)3/26/2013 10:30:34 AM - bestsync (166.70.54.154)> REST 9490432
(000016)3/26/2013 10:30:34 AM - bestsync (166.70.54.154)> 350 Rest supported. Restarting at 9490432
(000016)3/26/2013 10:30:34 AM - bestsync (166.70.54.154)> RETR /R-Disk/Mentors/Edelman, Rick/Podcasts/2013_03_16 TruthAboutMoneyRicEdelmanShow031613HourOne.mp3
(000016)3/26/2013 10:30:34 AM - bestsync (166.70.54.154)> 150 Connection accepted, restarting at offset 9490432
(000016)3/26/2013 10:30:43 AM - bestsync (166.70.54.154)> 426 Connection closed; transfer aborted.
(000016)3/26/2013 10:30:44 AM - bestsync (166.70.54.154)> PASV
(000016)3/26/2013 10:30:44 AM - bestsync (166.70.54.154)> 227 Entering Passive Mode (23,30,60,44,195,126)
(000016)3/26/2013 10:30:44 AM - bestsync (166.70.54.154)> REST 10846208
(000016)3/26/2013 10:30:44 AM - bestsync (166.70.54.154)> 350 Rest supported. Restarting at 10846208
(000016)3/26/2013 10:30:44 AM - bestsync (166.70.54.154)> RETR /R-Disk/Mentors/Edelman, Rick/Podcasts/2013_03_16 TruthAboutMoneyRicEdelmanShow031613HourOne.mp3
(000016)3/26/2013 10:30:44 AM - bestsync (166.70.54.154)> 150 Connection accepted, restarting at offset 10846208
(000016)3/26/2013 10:30:54 AM - bestsync (166.70.54.154)> 426 Connection closed; transfer aborted.
(000016)3/26/2013 10:30:54 AM - bestsync (166.70.54.154)> PASV
(000016)3/26/2013 10:30:54 AM - bestsync (166.70.54.154)> 227 Entering Passive Mode (23,30,60,44,195,127)
(000016)3/26/2013 10:30:55 AM - bestsync (166.70.54.154)> REST 12201984
(000016)3/26/2013 10:30:55 AM - bestsync (166.70.54.154)> 350 Rest supported. Restarting at 12201984
(000016)3/26/2013 10:30:55 AM - bestsync (166.70.54.154)> RETR /R-Disk/Mentors/Edelman, Rick/Podcasts/2013_03_16 TruthAboutMoneyRicEdelmanShow031613HourOne.mp3
(000016)3/26/2013 10:30:55 AM - bestsync (166.70.54.154)> 150 Connection accepted, restarting at offset 12201984
(000016)3/26/2013 10:31:04 AM - bestsync (166.70.54.154)> 426 Connection closed; transfer aborted.
(000016)3/26/2013 10:31:05 AM - bestsync (166.70.54.154)> PASV
(000016)3/26/2013 10:31:05 AM - bestsync (166.70.54.154)> 227 Entering Passive Mode (23,30,60,44,195,128)
(000016)3/26/2013 10:31:05 AM - bestsync (166.70.54.154)> REST 13557760
(000016)3/26/2013 10:31:05 AM - bestsync (166.70.54.154)> 350 Rest supported. Restarting at 13557760
(000016)3/26/2013 10:31:05 AM - bestsync (166.70.54.154)> RETR /R-Disk/Mentors/Edelman, Rick/Podcasts/2013_03_16 TruthAboutMoneyRicEdelmanShow031613HourOne.mp3
(000016)3/26/2013 10:31:05 AM - bestsync (166.70.54.154)> 150 Connection accepted, restarting at offset 13557760
(000016)3/26/2013 10:31:16 AM - bestsync (166.70.54.154)> 426 Connection closed; transfer aborted.
(000016)3/26/2013 10:31:17 AM - bestsync (166.70.54.154)> PASV
(000016)3/26/2013 10:31:17 AM - bestsync (166.70.54.154)> 227 Entering Passive Mode (23,30,60,44,195,129)
(000016)3/26/2013 10:31:17 AM - bestsync (166.70.54.154)> REST 14913536
(000016)3/26/2013 10:31:17 AM - bestsync (166.70.54.154)> 350 Rest supported. Restarting at 14913536
(000016)3/26/2013 10:31:18 AM - bestsync (166.70.54.154)> RETR /R-Disk/Mentors/Edelman, Rick/Podcasts/2013_03_16 TruthAboutMoneyRicEdelmanShow031613HourOne.mp3
(000016)3/26/2013 10:31:18 AM - bestsync (166.70.54.154)> 150 Connection accepted, restarting at offset 14913536
(000016)3/26/2013 10:31:28 AM - bestsync (166.70.54.154)> 226 Transfer OK

[botg]

What does "Inbound Layer 7" mean? What if you disable it?

[mcjathan]

What does "Inbound Layer 7" mean? What if you disable it?

I didn't know what what "Inbound Layer 7" is until you asked. I Googled it, and I now only have a vague idea. I did find a few Google pages that seemed to indicate that my problem might be fixed if I turned it off, but I also found some other pages said turning it off could shut-down incoming TCP communications.

The bottom line is that I turned it off (see the attached image) and it seems to have no noticeable effect:

1) Communications in and out seem to work exactly as before, and
2) The error "426 Connection closed; transfer aborted" is still happening pretty much the same. The disconnection interval this time seems to be about every 9 seconds.

Boco, what do you suggest now?

Regards, Jeff

[botg]

Router configuration looks good now. Maybe some firewall on either the client or server machine?

[boco]

Could you try an alternative client?

[mcjathan]

Could you try an alternative client?

This was a great suggestion, boco. I've not solved the problem, but your suggestion led me to some interesting (and hopefully helpful) observations:

1) I used the alternative FTP client WinSCP to download a large file (from the same Filezilla FTP server and used the same Filezilla server login). The large file downloaded flawlessly without any connection breaks and restarts.

2) However, as I was thinking about this, I realized that my WinSCP is configured to only download one file at a time while my original FTP client downloads multiple files concurrently. Perhaps the problem is due to multiple connections vs. a single connection?

3) I returned to my original FTP client (BestSync) and modified its configuration to only allow one connection at a time. Bingo! My original FTP client downloads flawlessly if I only allow one connection at a time.

4) Next, I returned to WinSCP and attempted to configure it to download multiple files concurrently, but I could not get WinSCP to download multiple files at once so I was unable to test my hypothesis with WinSCP.

5) So, I installed Filezilla Client (FC) and configured FC to download multiple files concurrently. FC downloaded multiple files concurrently without any disconnects and restarts. So, FC downloads multiple files concurrently but my original FTP client (BestSync) has the disconnect/reconnect problem.

6) Next, I noticed another possible clue. All of the data referenced in my previous postings in this thread is from my original client across our WAN to the Filezilla server. However, I also have the original FTP client (BestSync) installed on a machine that connects to the same Filezilla server on our LAN. In this case, my FTP client and the Filezilla server are 30 ft away and are connected by a 1 Gbps LAN. Also, in this case, the connection between the client and server bypasses at least three known routers (probably more). The client and server are actually connected only by an single managed switch. To my amazement, the disconnects and reconnects happened in this case. I noticed that the disconnect/reconnect happened very quickly (less than a second) instead of the 10 or 20 seconds I observed previously. This got me thinking that perhaps the disconnect/reconnect is not time dependent (time-outs) but instead depends on the amount of data transmitted.

7) So, I went back to the three Filezilla Server logs I saved and calculated the DIFFERENCE in the restart offsets. See the results in the image I've attached below. Some observations:

a) The three runs all had different offsets.
b) However, for a given run the offset was perfectly the same throughout the entire run.
c) The third run had the largest offset. The third run also had the fasted connection speed between the client and server. Is this a clue?

What do you think? Obviously, the Filezilla FTP client has no problem while my original FTP client (BestSync) does have a problem. What do my observations indicate could be the root problem?

[botg]

All offsets are multiples of 4096.

My guess is a lost socket event in your client's receive loop.

Try the following: Use Wireshark on the machine running the client and capture the traffic between two restarts. Calculate the restart offset difference as indicated by the REST command and compare it against the number of bytes actually received over the wire. If my earlier guess is right, it is more than the restart offset.

[mcjathan]

All offsets are multiples of 4096.

My guess is a lost socket event in your client's receive loop.

Try the following: Use Wireshark on the machine running the client and capture the traffic between two restarts. Calculate the restart offset difference as indicated by the REST command and compare it against the number of bytes actually received over the wire. If my earlier guess is right, it is more than the restart offset.

Thanks for that response, botg.

The debugging you describe is starting to go beyond my pay-grade ;^). However, I am planning to direct the developer of the FTP client (BestSync) to this thread so he can track down this problem. Is there anything else you can add to your suggestion of how to track this down?

Regards, Jeff

[botg]

Only speculation, I'm not familiar with that client.