Segmented Downloads - Any Plans?

Come here to discuss FileZilla and FTP in general

Moderator: Project members

Message
Author
User avatar
boco
Contributor
Posts: 25334
Joined: 2006-05-01 03:28
Location: Germany

Re: Segmented Downloads - Any Plans?

#31 Post by boco » 2012-11-26 16:02

In this thread (among others), it is repeatedly mentioned that FileZilla supports multiple simultaneous downloads. Presumably this means you can have more than one download from more than one server happening at the same time, since obviously this does not mean you can have multi-part segmented downloads.
Correct, up to ten at a time.
However, in this thread that I started asking the exact same question - how to do multiple simultaneous downloads - I am told that FileZilla cannot do this.
No. I just told you that there's one queue that all server tabs share, and files are downloaded sequentially. That means first all files from server 1, then server 2 etc. You cannot easily interleave them (downloading from different servers at the same time). The only influence you have on the sequence is defining priorities.
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Do yourself a favor and read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
### END SIGNATURE BLOCK ###

xeon
226 Transfer OK
Posts: 128
Joined: 2009-08-19 03:18

Re: Segmented Downloads - Any Plans?

#32 Post by xeon » 2012-11-26 21:20

bmn123 wrote:it's simply an extension of functionality onto a protocol that wasn't originally written to optimally handle single files of such size.
FTP handles large files just fine, segmented downloads are just a sloppy way to try and work around the real problems. (i.e. Low quality network stacks, FTP clients/servers, modems/routers, ISPs, etc.)

I personally make an effort to avoid all of these, and I have no problem getting several hundred Mbps across the entire US and the transatlantic with a single connection. Even on my residential connection, I have no problem maxing out the 50Mbit download under the same circumstances.

bmn123

Re: Segmented Downloads - Any Plans?

#33 Post by bmn123 » 2012-11-26 23:47

xeon wrote:
bmn123 wrote:it's simply an extension of functionality onto a protocol that wasn't originally written to optimally handle single files of such size.
FTP handles large files just fine, segmented downloads are just a sloppy way to try and work around the real problems. (i.e. Low quality network stacks, FTP clients/servers, modems/routers, ISPs, etc.)

I personally make an effort to avoid all of these, and I have no problem getting several hundred Mbps across the entire US and the transatlantic with a single connection. Even on my residential connection, I have no problem maxing out the 50Mbit download under the same circumstances.
I misspoke; I meant in the context of such problems. The fact of the matter is, there is a very large number of users that, for whatever reason, do not have an optimal setup. Segmented downloading would not have been developed and become popular if it had not been a good catch-all solution to such problems. I don't see anything sloppy about that.

Personally, I have an ISP that does not throttle FTP traffic, a properly configured VPS on Leaseweb-owned servers which AFAIK doesn't either, and a properly configured modem/router. That's about all I can reasonably do. I cannot do anything about network infrastructure or congestion that is out of my control. Likewise, there are many people who are don't even have as much control as I do -- they may be in countries that are farther away from their FTP server, possibly with infrastructure that is sub-par; they also may have limited ISP options due to monopolies or government regulations.

As topbanana said:
At the end of the day, the internet is not perfect. Servers might be busy or badly configured, internet backbones might be busy, the server might be on the other side of the world with many hops to get to it, some routers might give large latencies, ISPs might have crap networks, internet connections might not be perfect, etc... This is the real world. [...] All this add up to lots of us having slow FTP transfers.
If you can saturate a 100mb/s link with a single file transfer, then you probably don't see any of the above common real-world conditions, and are a very, very lucky guy. Unlike us. We just want to Filezilla to work at full speed in all weather conditions.

topbanana
450 Internal Error
Posts: 36
Joined: 2012-11-09 02:29

Re: Segmented Downloads - Any Plans?

#34 Post by topbanana » 2012-12-07 14:46

I tried to max out my connection with FileZilla the other day...

The way i managed to do it was to have two instances of FileZillla running, both downloading 10 concurrent files each...
I used WinSCP to login and split the big files i needed by creating a custom (remote) command of:
rar a -m0 -v20m "!?&Enter an Archive Name:?archive.rar!" !& (i.e. no compression, 20MB chunks)
This allows me to split the selected file(s) without needing to Putty in and faffing around with the command line... Simple-ish

Using just one instance with 10 concurrent downloads wasn't maxing out my connection very well on that day (15 hops to server, 330ms latency), so i had to run two instances. This gave a fairly flat trace on the Networking tab of Task Manager.
Using Free Download Manager to download some big files off of a HTTP server, straight away maxed out my 15Mbps connection with 12 segments... the trace was an exceptionally flat line...

Having to split, download, then delete the splits after was a achievable, but must surely be loading the server far more than just using a segmenting FTP client?! It loaded up my time for sure!

FTP just isn't that efficient with one transfer.

Edit: I was using SFTP, btw... Client is Filezilla, server is OpenSSH (not VSFTPD as i said earlier, it was installed, but obviously i wasn't connecting to it... now uninstalled)

User avatar
boco
Contributor
Posts: 25334
Joined: 2006-05-01 03:28
Location: Germany

Re: Segmented Downloads - Any Plans?

#35 Post by boco » 2012-12-07 15:53

Please note that the SFTP protocol is not particularly fast.
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Do yourself a favor and read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
### END SIGNATURE BLOCK ###

topbanana
450 Internal Error
Posts: 36
Joined: 2012-11-09 02:29

Re: Segmented Downloads - Any Plans?

#36 Post by topbanana » 2012-12-08 02:00

boco wrote:Please note that the SFTP protocol is not particularly fast.
Which is why we're asking for segmented downloads... As that makes it fast.

User avatar
botg
Site Admin
Posts: 33242
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: Segmented Downloads - Any Plans?

#37 Post by botg » 2012-12-08 09:12

Try the latest nightly from http://filezilla-project.org/nighly.php, it contains some improvements to SFTP performance.

topbanana
450 Internal Error
Posts: 36
Joined: 2012-11-09 02:29

Re: Segmented Downloads - Any Plans?

#38 Post by topbanana » 2012-12-08 11:26

botg wrote:Try the latest nightly from http://filezilla-project.org/nighly.php, it contains some improvements to SFTP performance.
Just installed...
Looking at the network throughput traces in task manage, there is no change...
FZ was downloading for a while before installing the nightly, then i restarted/resumed the transfers straight away.

What were the changes?

User avatar
boco
Contributor
Posts: 25334
Joined: 2006-05-01 03:28
Location: Germany

Re: Segmented Downloads - Any Plans?

#39 Post by boco » 2012-12-08 14:18

http://filezilla-project.org/changelog.php

Contains all changes made to the sources.
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Do yourself a favor and read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
### END SIGNATURE BLOCK ###

topbanana
450 Internal Error
Posts: 36
Joined: 2012-11-09 02:29

Re: Segmented Downloads - Any Plans?

#40 Post by topbanana » 2012-12-09 02:15

Perhaps those changes improve the performance when the network conditions are more ideal...

I'm still stuck at about 128KiB/s* for any single transfer...

* it varies quite a bit around this value...

xeon
226 Transfer OK
Posts: 128
Joined: 2009-08-19 03:18

Re: Segmented Downloads - Any Plans?

#41 Post by xeon » 2012-12-09 14:20

topbanana wrote:Perhaps those changes improve the performance when the network conditions are more ideal...

I'm still stuck at about 128KiB/s* for any single transfer...

* it varies quite a bit around this value...
Why not get to the root of the problem and talk to your ISP or the network provider of the server(s) in question?

topbanana
450 Internal Error
Posts: 36
Joined: 2012-11-09 02:29

Re: Segmented Downloads - Any Plans?

#42 Post by topbanana » 2012-12-09 15:27

Or I could lay my own fiber across the pacific too perhaps?

Most of us don't have any say in the infrastructure between us and the server.

This is a very common problem, that is easily fixed by running more than one transfer concurrently or segmenting single files.

Internet connection speeds have increased massively recently, and this now shows how incredibly slow Filezilla can be for single transfers when things are less than perfect in between...

We don't want Filezilla to be left behind. It's too good in every other way. Which is why we're asking for this.

User avatar
jharris1993
503 Bad sequence of commands
Posts: 21
Joined: 2012-11-18 17:49
First name: Jim (JR)
Contact:

Re: Segmented Downloads - Any Plans?

#43 Post by jharris1993 » 2012-12-09 15:28

xeon wrote:
topbanana wrote:Perhaps those changes improve the performance when the network conditions are more ideal...

I'm still stuck at about 128KiB/s* for any single transfer...

* it varies quite a bit around this value...
Why not get to the root of the problem and talk to your ISP or the network provider of the server(s) in question?
Sir,

Unfortunately, we often do not have that kind of control over our network connections, and the administrators for networks/servers located hundreds, or even thousands, of miles away are often impossible to reach. That's assuming that they even speak English.

Instead of continually shifting the blame to some other - nebulous - third party, why not just step up and help us do what we need to do.

In the time it has taken you to reply to all these postings, defending your position and pointing to everyone else's problems, you could have implemented the feature and added two or three dot-releases as well.

As many posters above have already said, you must be fortunate to have your internet connection connected directly to a low-latency gigabit Internet backbone. Likewise, your renown within the industry appears to allow you first-name access to even the most irascible server and ISP providers.

Unfortunately, the rest of us are mere mortals. We do not have God-like access or influence within the industry. As a consequence, we mere mortals are forced to deal with what we have, as pitiful as it may be.

So, instead of spending all this time and energy dodging a problem that clearly needs to be addressed, just step-up and provide us with a solution that will help solve this problem.

What say ye?

Jim (JR)
Some see things as they are, and ask "Why?"
I dream things that never were, and ask "Why Not".

Robert F. Kennedy

“Impossible” is only found in the dictionary of a fool.
Old Chinese Proverb

wayner
500 Command not understood
Posts: 3
Joined: 2012-12-10 13:55
First name: Wayne
Last name: Kozun

Re: Segmented Downloads - Any Plans?

#44 Post by wayner » 2012-12-10 14:28

I have a server in Europe and I often download files via FTP to my home in Canada. The fastest speed that I can get for FTP downloads is 2.1Mbps (=16.8mbps) - so for some reason either my server, my ISP in Canada or somewhere along the line the FTPs are being throttled to this speed which is only a fraction of my 150 mbps download speed. But I can download multiple files simultaneously at this speed with Filezilla and other clients.

CuteFTP Pro allows multipart downloads which splits up a file into 8 pieces and downloads the 8 pieces as separate files and then recombines them. This speeds up downloads by almost 8X, it takes a bit of time to recombine the files when the download(s) is/are completed.

Therefore the time to download a 1 GB file in Filezilla is 8 minutes, in CuteFTPPro it is 1.5 minutes.

Why doesn't Filezilla support Multipart downloads?

I really don't find your responses in this thread very compelling. I have zero power to have my ISP change how they do things.

I really like using Filezilla and I am comfortable with it, but this is a very useful feature and it would be great to see it in Filezilla.

topbanana
450 Internal Error
Posts: 36
Joined: 2012-11-09 02:29

Re: Segmented Downloads - Any Plans?

#45 Post by topbanana » 2012-12-11 02:10

Question: Could multipart/segmented transfers work better with SFTP? (rather than plain FTP)

I understand it is quite different under the hood?

Locked