Mysterious Filezilla 0.925 Server Hang

[thedfox]

I have had a FZ Server in production for some time. The server works great with the exception of one issue that I am having with it. Here is some background info about the configuration:

- It is configured to use SSL(explicit) and to accept connections on port 990.
- It has around 200 ports available for outbound connections from the server for the passive mode custom port range.
- Windows 2003 Server R2 with Dual 3Ghz procs and 2Gb of RAM. U320 SCSI Raid1 set.


Here is some info regarding the symptom:

The server can service numerous connections (50000+), when at any time it will hang. When the server encounters this problem, the filezilla server service still appears to be running and the server is still listening on port 990, but will not accept new connections. If the admin console is open during this time any open connections in the console appear to freeze. When this problem occurs I also check to make sure that I have not used up the outbound custom port range as well. At this point you can stop the service, but it take a long time and does not stop successfully. So I usually kill the process and restart the service until the server hangs again. I have run this configuration on at least three different servers and have had the same thing happen. I have also been seeing this issue since at least 0.9.22.

Any advise or assistance would be greatly appreciated.

Thanks,

[botg]

Are you using any firewalls and virus scanners? What happens if you completely uninstall them? Note: Don't just disable them, that leaves their drivers running and causing havoc. Only way to fully disable virus scanners and firewalls is to completely uninstall them.

[thedfox]

No firewall is installed on the machine. Antivirus was installed but disabled. I will completely remove it and give it another go.

[thedfox]

The antivirus has been removed and the problem still occurs. The server is nat'd behind a firewall, and the custom port range is allowed in the firewall.

[botg]

All Windows patches installed? All drivers, especially for the network card using bleeding edge versions? Make sure the Application Gateway Layer Service (ALG.exe) is not running.

[thedfox]

Yes. ALG is not running. 2003 R2 with SP2. TCP Chimney is also disabled. No hyperthreading either....

[botg]

Can you please run a network sniffer like for example Wireshark and log the TCP traffic when a client tries to connect after the server appears to hang?

[thedfox]

I ran the trace. As a result of the trace I went ahead and disable tx/rx checksum offload on the NIC. I have also changed the FZ config to only listen to the external interface instead of *. Now to test.....

[thedfox]

I have encountered the issue again. I ran a wireshark capture after the problem occurred. No packets were captured.

[thedfox]

I did this once more and now have some capture data. (Must not have chosen the correct filter the first time). I see a SYN->SYN->ACK->RST to the server. The RST packets have the following for the Acknowledgment number: Broken TCP. The acknowledge field is nonzero while the ACK flag is not set. I have packet capture if you would like to see it. Thanks Again.

[MikeySea]

Bump.

So whatever came of this issue. Any conclusions with the sniffer trace?

We are seeing a very similar / almost identical issue with using 0.9.27beta on our Win2K3 R2, latest drivers, etc. production systems.

Looks like we are going to have to put in a daemon/service to monitor the FZS and kill / restart it when this issue occurs.

[thedfox]

No Resolution. I had to do what you are speaking of. Implement a Watchdog that actually tries to login to the server. If it fails it will kill the FZ Service and restart it. I could not use a service monitor as the service still appeared to be in the started state when it was hanging. If you have any luck please post. Thanks

[MikeySea]

No Resolution. I had to do what you are speaking of. Implement a Watchdog that actually tries to login to the server. If it fails it will kill the FZ Service and restart it. I could not use a service monitor as the service still appeared to be in the started state when it was hanging. If you have any luck please post. Thanks

Okay - thanks for the quick followup and the additional details!

It may be a few weeks or more until we actually get something working to do this, but one thing for sure is everyone here wants the timeout/heartbeat value to be configurable so that false alarms can be remedied in the field if needed Something along the lines of 30 to 60 seconds or so (experimentation needed). Do you have a set time?

Also, dual approach may also be considered. Something that monitors an already existing connection's health/responsiveness as well as the accepting of new connections.

[thedfox]

It checks on a 5 Minute interval now. I have a FZ User created just for this purpose. If login fails, then a batch file is called that does a pskill, restarts the FZ service, and sends me an email to let me know that it has done such.