Upload speeds & TCP window size

[MrSanchez]

Hello,

This is mostly a desperate request for aid with an issue that has been bothering me for a week now. I couldn't express the amount of appreciation I'd have if someone could help me.

I've been creating my first GUI application using Qt, coded in C++, it's been going well, Google's been a great help.
I've been able to connect my application to a FTP server with both downloading & uploading capabilities. I mostly rely on an old Qt class called QtFTP for the handling of this.
The program I'm making is not a FTP client like FileZilla or anything, it just uses FTP.

My problem is the upload speed of files using my program in comparison to FileZilla's upload speed. It seems that my program won't upload with a speed faster than 240kbps. I've calculated this by the fact it takes exactly 90 seconds to upload a 21.55mb file.
I've been desperately looking for a fix because FileZilla manages to upload the same file with a speed up to 1.3mb/s. It's worth noting I have a 150/15mb dwn/up connection and I use Windows 7.

I downloaded WireShark to observe the packets and noticed a difference in the way packets were sent.
The FTP-data messages during the upload of my program reported a size that, slow-start wise would be small at the beginning (~3000 bytes) but would very rapidly turn into fixed messages of precisely 16384 bytes til the end of the transfer, alike: https://gyazo.com/5e5d22eddc692e8d891b3ba1de27fe07
Observing the packets of FileZilla showed that the FTP-data messages had a varying size, alike: https://gyazo.com/f42c87a2dd6968028cf78a7dee0b866e

I also looked at the threeway-handshake because after some Google research I found out about window sizes and how they could affect bandwidth. Because of this I also found an old post in which botg explains how to calculate max bandwidth: viewtopic.php?t=32591

Because of this I calculated that my RTT is: 128,697ms
And according to this handshake my window scale is 4: https://gyazo.com/934ed1425df3d2ed7fdfd001adf5a6ad

However, that thread describes downloading speeds, not uploading, but I assume the max window is still bottlenecked by the client, so still 4*65535=262140
Max bandwidth = 262140 / 128,697 = 2.04MiB/s = 254KB/s (surprisingly close to what my max upload speed seems to be).

FileZilla's handshake: https://gyazo.com/6f91543c554c50b03aac08628c99fbbf

One of the things I don't understand is how the window size value in the SYN packets is proposed or changed. My program shows Win=8192 in the SYN packet whereas FileZilla shows Win=65535 even after I tried changing the recv & send (v2) buffer sizes to random high-low values in Filezilla.xml. It always stayed as 65535. Changing the values did not seem to make any major effect upon the upload speed of the files via FileZilla either.

In my application I've even tried to set the send buffer at various stages of the socket class, pre-connection (gave 10038 error) and post connection (no result) using the following code:

Code: Select all

    qint64 send_buffer_size = 0; // 0 was the last desperate thing I tried. I've set it to 4096, 65535, 512*1024 etc...
    int send_err = ::setsockopt(socket->socketDescriptor(), SOL_SOCKET, SO_SNDBUF, (char *)&send_buffer_size, (qint64)sizeof(send_buffer_size));
    int err = WSAGetLastError();

All in all everything I've tried to get my application to have the same upload speed as FileZilla has failed. Nor could I affect FileZilla's upload speed with the aforementioned buffers.
How does FileZilla cause for a higher upload speed while uploading the same files to the same server?

Is there anyone who could please help me? I have ran out of ideas.

Kind regards,
Sanchez

[botg]

The FTP-data messages during the upload of my program reported a size that, slow-start wise would be small at the beginning (~3000 bytes) but would very rapidly turn into fixed messages of precisely 16384 bytes til the end of the transfer, alike: https://gyazo.com/5e5d22eddc692e8d891b3ba1de27fe07

Very regular. The window sizes aren't the problem, it looks like your program simply isn't sending data fast enough.

In my application I've even tried to set the send buffer at various stages of the socket class, pre-connection (gave 10038 error) and post connection (no result) using the following code:

10038 is WSAENOTSOCK. To set the buffer size pre-connection the socket obviously needs to be created first.

[MrSanchez]

Very regular. The window sizes aren't the problem, it looks like your program simply isn't sending data fast enough.

Wow..

At first I thought your answer wasn't going to help me much but I actually realized that I've been so tunnel visioned with the window sizes that I more or less neglected the other parts of FTP transfer.
There were 3 things that I did to get it working.

First of all, foolish me, after I spent several days with that QtFTP class I switched half of my code (only the uploading part) to a newer but more limited Qt class that allows FTP uploading only, to see if it would be better....meanwhile I was modifying the socket window sizes within the QtFTP class but turns out that code was never called for the actual socket that was uploading files (that was a different class). So in the end my window sizes never changed for uploading.

Secondly, due to your answer I looked away from window sizes and looked at the writing data class only to see that the blocksize and the read/write buffer were 16*1024...Changing this value made a huge impact.

Thirdly after I went back to using the old class I actually managed to change the window size now.

At the moment my send buffer size = 512 * 1024 and my read/write buf&blocksize are both 64*1024.
I'm not sure if there are any downsides to having huge buffer sizes like these but it is uploading my file now within 17 seconds.

FileZilla actually takes 23 seconds to upload the same file.
Not sure why because I set my XML settings back to what I think were the defaults:

Code: Select all

<Setting name="Socket recv buffer size (v2)">4194304</Setting>
<Setting name="Socket send buffer size (v2)">262144</Setting>

Either way it's no biggie. I'm very happy I can finally move on from reading Wireshark packets the whole week.
Thanks a ton, botg.

Kind regards,
Sanchez

[botg]

I'm not sure if there are any downsides to having huge buffer sizes like these but it is uploading my file now within 17 seconds.

FileZilla actually takes 23 seconds to upload the same file.

Which version of FileZilla are you using?

[MrSanchez]

I'm not sure if there are any downsides to having huge buffer sizes like these but it is uploading my file now within 17 seconds.

FileZilla actually takes 23 seconds to upload the same file.

Which version of FileZilla are you using?

Apparently 3.14.1 which seems pretty outdated....I swear I updated FileZilla at least once this year....I'll update and do another test.

edit: Updated to latest version, took 20 seconds this time. Still pretty good in my opinion.

edit2: After reuploading the same file 2x the download speeds of FileZilla spiked up and it uploaded the file in 16 seconds I'd say no problem with FileZilla, it's still awesome