ECONNABORTED during transfer of files

[audiopro]

I have an issue where I am downloading a batch of around 1500 files from a remote server to my local machine.
The transfer starts OK and over 1000 files are transferred without problems and then the transfer stops.
Further investigation reveals that the server has locked me out due to having too many connections - usually between (400 and 500).
I am using plain FTP and have tried both passive and active modes.
My ISP have blamed everybody and his dog for this error but I feel the issue is simply a case of some connections not being closed for some reason. Most of the connections must be being closed otherwise it would fail after transferring 400 - 500 files.

I have max # of connections set to 1.

The transfer report offers no clues but I understand why the server closes down but that is only as a result of file handles not being tidied up, can anyone offer any suggestions as to why this is happening.

Code: Select all

Status:    Starting download of xxxxxx/xxxxxx/php/images/pastelswhite/images/pastelswhite1.png
Status:    File transfer successful, transferred 930 bytes in 1 second
Status:    Starting download of xxxxxx/xxxxxx/php/images/pastelsyellow/pastelsyellow.psd
Command:    CWD xxxxxx/xxxxxx/php/images/pastelsyellow
Response:    250 OK. Current directory is xxxxxx/xxxxxx/php/images/pastelsyellow
Command:    PWD
Response:    257 "xxxxxx/xxxxxx/php/images/pastelsyellow" is your current location
Command:    PORT xxx.xxx.xxx.xxx
Response:    200 PORT command successful
Command:    RETR pastelsyellow.psd
Error:    Disconnected from server: ECONNABORTED - Connection aborted
Error:    File transfer failed
Status:    Disconnected from server
Status:    Connecting to xxx.xxx.xxx.xxx...
Status:    Connection attempt failed with "ETIMEDOUT - Connection attempt timed out".
Error:    Could not connect to server
Status:    Disconnected from server
Status:    Delaying connection for 5 seconds due to previously failed connection attempt...
Status:    Connecting to xxx.xxx.xxx.xxx...

[botg]

Possibly a firewall. Since every FTP transfer involves a new connection, transferring many files confuses some badly written firewalls into thinking they are under attack.

[audiopro]

I have tried it with the router and Windows firewalls both on and off and that makes no difference, could it be the firewall on the server?
Could the firewall be interrupting some of the commands to close connections?

[botg]

Yes, a server-side firewall could cause this.

[audiopro]

My ISP keep asking for screen shots of all the Filezilla settings but as I can download large batches of files without issues, I am assuming my settings are correct.
I can't think of any setting which would cause this issue on random files.
Could you suggest a way of locating this annoying error.
The 5 minute delay in uploading the files, I could cope with but my server locking me out for 5 minutes is intrusive.

[botg]

Do you only experience this issue with this one server or also with other servers hosted by a different companies?

If it only happens on this one server it's most likely a server-side issue. If it happens everywhere the problem is on your end.



Another thing you can try is setting a speed limit in FileZilla so that fewer transfers are performed per time unit. If this makes the problem disappear, the firewall-thinks-its-an-attack hypothesis gets strengthened considerably.

[audiopro]

Thanks for the response.
I use other servers but I don't tend to download as many files on the others.

I will try the speed limit to see if it makes a difference.
I don't mind waiting a bit longer for the transfers to take place but I can't be locked out of the server - it stops me working.

It may not be connected but if I refresh the index page of that particular site a few times in succession, then the server locks me out then too, with the same problem of too many connections. I assume a connection is made each time a picture is loaded but there are only about 40 pictures on the page so I have no idea what ramps up the connection count and worryingly, neither does the ISP.

[boco]

Further investigation reveals that the server has locked me out due to having too many connections - usually between (400 and 500).

Strong indication that connections aren't closed properly (or with delay). Since the server measures the number of connections from its end, it hints to an issue server-side - FileZilla will definitely not open that many connections on purpose.

[audiopro]

Had an interesting session with one of the support people today.
After some poking around we now know that after the page requesting 50 or so pictures has loaded, it takes up to 2 minutes for the connections count to return to single figures. This just seems totally wrong to me as I would expect the connection to close straight after the files has been served.
The tech told me that there is a 'keep alive' command which keeps the connection open in case it is needed again. Why is this if each connection is only used once and then discarded?
Sorry if this isn't Filezilla related but it may help someone else in a similar position.

[botg]

Keep-alive commands cannot cause this. This is the result of a bad firewall or NAT router.

[audiopro]

Something is keeping the connections open, if not keep alive commands then what?

I have keep alive turned off in Filzilla.

[botg]

A broken firewall or NAT router comes to mind.

[boco]

Connections are closed through sending a special packet and receiving the reply to it.
If a firewall/router blocks either the FIN packet and/or the reply without notifying both endpoints properly, the connection stays open until it times out, either completely or on one side (half-open connection).
Idle connection may be closed after ~2 hours at least, however that standard is violated very frequently.

I know it doesn't help with the problem.

[audiopro]

Thanks for taking the time to consider this.
What I can't get my inexperienced brain round I why would it be just the close/confirmation signals that were being lost?
By the same token, if the router/firewall was blocking them, I thought it would block all of them rather than a random selection.

[botg]

The special packet is FIN. See http://www.tcpipguide.com/free/t_TCPCon ... tion-2.htm for a short overview on how closing TCP sockets works.

why would it be just the close/confirmation signals that were being lost?
By the same token, if the router/firewall was blocking them, I thought it would block all of them rather than a random selection.

When a TCP connection is first established, at the initial SYN packet, a NAT router adds a new entry in its NAT table. Likewise, a stateful firewall employing connection-tracking starts tracking the connection with the SYN. At the end of the connection, these entries eventually get removed. If they get removed too early, then the last few packets of a connection can be dropped.

When this happens, the endpoints may actively send FIN packets and wait for the ACK to close the TCP connection, yet the NAT router or firewall drop these packets. It is only after some minutes that the TCP connections eventually time out and get ungracefully reset.