FileZilla Forums

Hi, it appears the server limits the value one can enter for 'socket buffer size'. For a trans-oceanic link of >250ms RTT and Gig internet connection both ends, BDP is:

(1,000,000,000 / 8) * 0.25 = 31MB.

Suggest allowing user to enter values of at least 100MB, to accomodate >500ms and 10G links.

Also if the user does not enter a value, suggest letting the system-defined socket defaults apply (i.e do not call setsockopt SO_SNDBUF with arbitrary 64K) (I recall reading this was always set but I may have mis-read).

btw Overly large/naive values should not cause problems since all OS's defaults (and sane sysadmin) apply their own max value (as should be seen in getsockopt()), e.g net.ipv4.tcp_wmem, net.ipv4.tcp_rmem etc

Regards,

ken

Suggest allowing user to enter values of at least 100MB, to accomodate >500ms and 10G links.

Upper limit will be set to 1G in the next version.

Also if the user does not enter a value, suggest letting the system-defined socket defaults apply (i.e do not call setsockopt SO_SNDBUF with arbitrary 64K) (I recall reading this was always set but I may have mis-read).

Experience has shown that the default value is in almost all cases too low and things like window scaling stay disabled if not explicitly setting the SND/RCVBUF options.

I believe on all recent Unix systems (mac Lin BSD Solaris), newer Windows too, window scaling is always enabled (if remote side also supports it), and a call to setsockopts is not needed. Unfortunately any 'factory' default is set, will only reduce (never increase) the window/buffer size that the sysadmin has set, or that the OS is using at any given moment (based on its detection of available memory).

Since setting SO_SNDBUF is only a request and the OS will pick the smaller of its administrator-configured or OS-calculated limits and SO_SNDBUF, the only effective use of setting a default SO_SNDBUF is it to a very large value; which is the same effect as not setting it. Otherwise the default value will set it lower that what the OS/sysadmin determined optimal.

Since setting SO_SNDBUF is only a request and the OS will pick the smaller of its administrator-configured or OS-calculated limits and SO_SNDBUF, the only effective use of setting a default SO_SNDBUF is it to a very large value; which is the same effect as not setting it. Otherwise the default value will set it lower that what the OS/sysadmin determined optimal.

On the same grounds of reasoning, if SO_SNDBUF is only a request, could the OS pick a larger number of its administrator-configured lower limit if the given value is too small?


Furthermore, a too large buffer size is also problematic, easily leading to transfer timeouts as the system's TCP stack will not inform the program that it can send more data because there is still a buffer to be processed, with program only being notified once buffer fill level falls below a certain level.
Huge buffer, very slow connection, long amount of time where application doesn't see anything happen at all from its point of view -> timeout.

On the same grounds of reasoning, if SO_SNDBUF is only a request, could the OS pick a larger number of its administrator-configured lower limit if the given value is too small?

No, verifiable by getsockopts(SO_SNDBUF) after the connection is opened. Setting SO_SNDBUF (also RCV) is in effect saying "dont use more than this".

Historically (before window scaling) its behavior may have allowed increase over the OS default, but only in the range under 65536 B (I'm guessing). Nowadays it might be argued SO_SNDBUF is most useful for sending, to set a maximum number of packets in-flight so the sender can insure the end of a stream was received (if flush() returns before the ack of the last byte), or simple form of throttling without userspace waits, or apps designed to estimate network throughput sending octet streams who's length is <= the size of SO_SNDBUF (the buffer would consume the entire length in very little time).

too large buffer size is also problematic, easily leading to transfer timeouts as the system's TCP stack will not inform the program that it can send more data because there is still a buffer to be processed, with program only being notified once buffer fill level falls below a certain level. Huge buffer, very slow connection, long amount of time where application doesn't see anything happen at all from its point of view -> timeout.

for the case where network is fast but the available bandwidth fluctuates, yes. i.e the windowing system detects high bandwidth and provides a larger buffer, which consume all the app bytes, and then if there is sudden high contention (low throughput) on the link, those in-buffer or in-flight packets create the effect of a write timeout. This is where a app or OS defined small timeout might be helpful, but arguably shouldn't be the rule/default. Some may prefer to increase their write timeout, vs using a smaller SO_SNDBUF (one manually set and not auto by the tcp stack), to have higher overall throughput.

TCP has no built-in application layer timeout, if all packets have been acked a connection can stay idle yet open indefinitley.

This is especially problematic for downloads in combination with broken routers and firewalls wich silently drop connections without sending RST to both peers. Connection seems completely idle, no data arriving, yet the local peer still thinks the connection is open and dutifully waits for data.

For this very reason, application layer timeouts are vital and cannot be eliminated.


Regarding flush() and similar calls, consider that FileZilla like all good GUI programs use non-blocking IO for obvious reasons. Flush is inherently blocking and thus cannot be used.