Segmented Downloads - Any Plans?

Come here to discuss FileZilla and FTP in general

Moderator: Project members

Message
Author
topbanana
450 Internal Error
Posts: 36
Joined: 2012-11-09 02:29

Re: Segmented Downloads - Any Plans?

#61 Post by topbanana » 2012-12-22 11:34

botg wrote:If the network is congested, how does using multiple parallel transfers make the congestion go away?
What if the route has high latency (which probably is a symptom of congestion too), like i have being on the other side of the world from my server.

I'm assuming that transferring data via FTP involves the client asking for a packet(s) of data, it then being sent by the server, then the client acknowledges it's got it, then asks for the next bit, and so on... (Please do correct me here, i'd like to know... a quick google didn't come up with the juicy details... is it just TCP?)

If the latency is high, the time spent sitting there waiting for the data to start flowing, which may itself come down nice and quick, and then time waiting for the acknowledgement to get back and the next packet sent...
The proportion of time sitting around idle is higher...
So if we send off for two (or more) packets at once, we land up waiting less time for more date...

You mentioned the small file issue, the need for multi-threaded transfers... Isn't this really the same issue? But on a different scale???

Again, since us clients now have nice big fast internet connections, these issues become very visible. And they're only going to get worse for us poor users.


We can all prove this hypothesis very easily. So easily and reliably in fact, that i'd call it a theory.


If you can't see what we're getting at, then you should do some real-world tests. Connect to some servers on the other side of the world. Or next time you get out of Köln and go somewhere a bit more 'back country', internet wise, take your laptop with you. Please.


Filezilla rocks, in every other way...

User avatar
botg
Site Admin
Posts: 33236
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: Segmented Downloads - Any Plans?

#62 Post by botg » 2012-12-22 20:54

In its default configuration, FileZilla manages around 3.6 MiB/s at a latency of one full second. Is your desired bandwidth delay product larger than 4194304 bytes?

topbanana
450 Internal Error
Posts: 36
Joined: 2012-11-09 02:29

Re: Segmented Downloads - Any Plans?

#63 Post by topbanana » 2012-12-22 22:59

botg wrote:In its default configuration, FileZilla manages around 3.6 MiB/s at a latency of one full second. Is your desired bandwidth delay product larger than 4194304 bytes?
How do you measure it?

What would be your throughput if you downloaded 10 files at once over that connection?
(What was you internet connection's speed? How many hops to the server?)


Do you not believe us that our transfers are as slow as we say?
Do you not believe us when we say that if we transfer 10 chunks/files simultaneously, out throughput goes up 10 fold(ish), or up to saturation?
Do you not understand that most people on this planet don't have any say whatsoever in the network route between us and our servers?
Do you not believe us when we say that when trying other clients that do multi-part, our throughput reliably jumps up to saturation (but those clients are ugly)
Do you not believe that many people in the world suffer from FTPs 'notorious' slow speeds?

User avatar
boco
Contributor
Posts: 25330
Joined: 2006-05-01 03:28
Location: Germany

Re: Segmented Downloads - Any Plans?

#64 Post by boco » 2012-12-23 04:33

### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Do yourself a favor and read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
### END SIGNATURE BLOCK ###

topbanana
450 Internal Error
Posts: 36
Joined: 2012-11-09 02:29

Re: Segmented Downloads - Any Plans?

#65 Post by topbanana » 2012-12-23 04:38

Cheers, I'll have a go when i get back...

User avatar
botg
Site Admin
Posts: 33236
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: Segmented Downloads - Any Plans?

#66 Post by botg » 2012-12-23 09:09

topbanana wrote:What would be your throughput if you downloaded 10 files at once over that connection?
Compared to a single file, same total, a tenth on each individual transfer. That was transferring data from RAM. If using multiple transfers with files stored on ordinary rotating disks, total speed was always slower compared to a single transfer.
(What was you internet connection's speed? How many hops to the server?)
100MBit at that time. Been running tests over a transatlantic connection, from central Germany to west coast of the USA. Around 20 hops, give or take a few. Over that I managed 11MiB/s.
Do you not believe us that our transfers are as slow as we say?
Do you not believe us when we say that if we transfer 10 chunks/files simultaneously, out throughput goes up 10 fold(ish), or up to saturation?
Do you not believe us when we say that when trying other clients that do multi-part, our throughput reliably jumps up to saturation (but those clients are ugly)
I do believe you, but segmented downloads are not the solution, they are a workaround a best. In the context of FTP, they carry a high overhead and lack of protocol support induces a high chance of corrupt transfers. Ultimately if everyone would use segmented transfers, the situation would be exactly the same as before if not worse.
Do you not believe that many people in the world suffer from FTPs 'notorious' slow speeds?
Nice attempt, but your question is invalid. In the context of transfer speeds, FTP isn't slow at all, FTP is perfectly capable of saturating even the fastest links.
Do you not understand that most people on this planet don't have any say whatsoever in the network route between us and our servers?
Unless you get your Internet connection for free, you can always stop giving money to bad providers.

bmn123

Re: Segmented Downloads - Any Plans?

#67 Post by bmn123 » 2013-01-02 00:07

botg wrote:
topbanana wrote:Do you not understand that most people on this planet don't have any say whatsoever in the network route between us and our servers?
Unless you get your Internet connection for free, you can always stop giving money to bad providers.
It's not always that easy. Two examples:
  • people in the US who are in an area where there is only one crappy ISP (like Comcast) due to a monopoly
  • people who are in a part of Australia where there are many ISPs, but they all rent their lines from the same telco who owns all the copper (like Telstra)
In any case, that's not topbanana's point. You could have the best ISP in the world and they still don't necessarily have any control whatsoever over the congested part of the network route that is effecting your download speeds.
botg wrote:Ultimately if everyone would use segmented transfers, the situation would be exactly the same as before if not worse.
I don't see how this would happen -- right now, download bandwidth is not being saturated. How would everyone using segmented transfers (which does saturate people's download bandwidth) slow everything down? :?

topbanana
450 Internal Error
Posts: 36
Joined: 2012-11-09 02:29

Re: Segmented Downloads - Any Plans?

#68 Post by topbanana » 2013-01-02 02:36

botg wrote:Ultimately if everyone would use segmented transfers, the situation would be exactly the same as before if not worse.
Hardly anyone uses FTP in the grand scheme of things... Most of the world's traffic uses is HTTP, YouTube and bittorrent... Not FTP.
If you implemented segmented transfers in Filezilla, the 'few' of us in the world that use it, would just get our work done quicker. The same amount of data would be transferred, it would just be done quicker, using the throughput us end users are paying lots of money for.

User avatar
botg
Site Admin
Posts: 33236
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: Segmented Downloads - Any Plans?

#69 Post by botg » 2013-01-02 08:11

Congestion = Line is full. If you achieve higher speeds with segmented downloads, someone else will get slower speeds. Overall speed across all users always stays the same. If all use segmented downloads, all will at best have the same speed as before, but in reality fewer due to the added overhead.

topbanana
450 Internal Error
Posts: 36
Joined: 2012-11-09 02:29

Re: Segmented Downloads - Any Plans?

#70 Post by topbanana » 2013-01-02 08:26

botg wrote:Congestion = Line is full. If you achieve higher speeds with segmented downloads, someone else will get slower speeds. Overall speed across all users always stays the same. If all use segmented downloads, all will at best have the same speed as before, but in reality fewer due to the added overhead.
How about throttling... Segmented/multithreaded downloads defeat this very well.

How much overhead will there be if we're downloading a 1GB file? This overhead is surely legacy from the days of old when we had smaller files?

And once again. FTP is only a tiny fraction of the traffic on the internet. Us getting our files quicker, freeing up the network quicker isn't going to be noticeable by most users/admins on the net. Again, we're not downloading any more data, just the same amount quicker. If any particular network is congested a lot, then that's the signal to the admins/owners to upgrade it.

So you're happy to tell us to suck it up, to suffer slow transfers, because the kids are watching youtube and downloading movies?

We want the best FTP client to be as fast as the ugly ones.

wayner
500 Command not understood
Posts: 3
Joined: 2012-12-10 13:55
First name: Wayne
Last name: Kozun

Re: Segmented Downloads - Any Plans?

#71 Post by wayner » 2013-01-07 19:03

Forget Bittoreent, the biggest user now appears to be Netflix - it accounts for about 1/3 of US internet traffic. And video/audio streaming accounts for 58% of internet bandwidth usage at least in North America.

Image

User avatar
botg
Site Admin
Posts: 33236
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: Segmented Downloads - Any Plans?

#72 Post by botg » 2013-01-07 20:56

Net neutrality, all packets should be treated equally; things like throttling or treating certain types of traffic special just violate it. Net neutrality would be a blessing for the Internet and its users, but corporate lobby groups are preventing it in the interest of short-sighted profit.

bmn123

Re: Segmented Downloads - Any Plans?

#73 Post by bmn123 » 2013-01-30 23:11

botg wrote:Net neutrality, all packets should be treated equally; things like throttling or treating certain types of traffic special just violate it. Net neutrality would be a blessing for the Internet and its users, but corporate lobby groups are preventing it in the interest of short-sighted profit.
I think everyone on here would heartily agree that net neutrality would be a boon for the Internet -- but again, that's beside the point. The reality is that throttling -- and its attendant problems -- exist.

topbanana
450 Internal Error
Posts: 36
Joined: 2012-11-09 02:29

Re: Segmented Downloads - Any Plans?

#74 Post by topbanana » 2013-01-31 00:44

bmn123 wrote:
botg wrote:Net neutrality, all packets should be treated equally; things like throttling or treating certain types of traffic special just violate it. Net neutrality would be a blessing for the Internet and its users, but corporate lobby groups are preventing it in the interest of short-sighted profit.
I think everyone on here would heartily agree that net neutrality would be a boon for the Internet -- but again, that's beside the point. The reality is that throttling -- and its attendant problems -- exist.
I heartily agree.
And all the problems listed in this thread, and many more we haven't listed, also exist and cause us varying degrees of 'slow' transfers...

Almost all easily fixed by multi-threaded/multi-part/segmented downloads...

bmn123

Re: Segmented Downloads - Any Plans?

#75 Post by bmn123 » 2013-03-03 03:14

What would it take to add this feature on in a fork of FileZilla? Is that really the only solution? It still seems like adding it as a default-disabled, "experimental" feature, with a "not supported by FTP protocol -- may result in corrupt files!" warning, is best.

Locked