Filezilla freezes on large number of files

[abeconnally]

I am trying to download a directory from my server with over 850,000 files in it. The total size of the directory is about 50 GB. The transfer starts fine, but after around 300,000 files transferred, filezilla bogs down and the transfers just timeout. I have tried over and over, and the same thing happens every time.

The files themselves are small files, they are emails. Typically 100 KB or less.

I've tried turning off every program on my computer except filezilla, thinking it may be due to system resources. I have also turned off the message log and transfer queue on filezilla, again, thinking it might help with system resources.

But none of that works. Around 530,000 files left, filezilla just slows to a crawl and barely anything happens.

I'm using Filezilla 3.65.0 on MacOS.

[botg]

Very strange. Are you using any firewalls or AV software? What does it say in Activity Monitor about CPU consumption and memory usage of FileZilla?

[abeconnally]

No firewalls, no AV

Nothing crazy on AM about resource use

[abeconnally]

Very strange. Are you using any firewalls or AV software? What does it say in Activity Monitor about CPU consumption and memory usage of FileZilla?

AM showing filezilla using 900+ MB of memory, 16-19% cpu

Screen Shot 2023-08-31 at 7.55.07 PM.png (44.02 KiB) Viewed 2703 times

Screen Shot 2023-08-31 at 7.53.09 PM.png (23.75 KiB) Viewed 2703 times

And right now, it's just skipping files, not even transferring them and going super slow, several minutes per file.

[abeconnally]

The actual transfers go fast, but it'll take a minute to go to the next file, so it's nothing to do with connection speed, it seems to be some sort of bogging down in the rate filezilla manages the queue internally

[abeconnally]

I just timed it, set the transfers to do only 1 at a time, and it did 15 files, all below 50kb, in 10 minutes.

Memory use is around 950 MB

Disk use is interesting, 20GB read:

Screen Shot 2023-08-31 at 8.38.43 PM.png (44.36 KiB) Viewed 2661 times

[abeconnally]

At this rate, it will take about 245 DAYS to transfer 30GB

The strange thing is that 2 weeks ago, I downloaded the same directory, and it took about 5 hours. So, I know it can work, I just don't know why it's not now.

[abeconnally]

I am using sftp, not sure if that matters

[botg]

20GB bytes read from disk and 2.2MB written when in the process of downloading? That makes no sense at all.

Might hire an exorcist, or try voodoo...

[botg]

Just downloaded 2 million files over SFTP on macOS, not a single issue. I am not able to reproduce the issue you're having.

[abeconnally]

20GB bytes read from disk and 2.2MB written when in the process of downloading? That makes no sense at all.

Might hire an exorcist, or try voodoo...

I assume that is reading the directory that I am downloading to, it is 50 GB in total, 20 GB downloaded so far, 30GB to go

[abeconnally]

Just downloaded 2 million files over SFTP on macOS, not a single issue. I am not able to reproduce the issue you're having.

I know, it's really weird. I have downloaded this directory before, without issue. But now, I can't seem to do it.

Any tips? Maybe clearing out my history or something?

[botg]

Overheating or broken storage device? Other than that I'm out of ideas.

I assume that is reading the directory that I am downloading to, it is 50 GB in total, 20 GB downloaded so far, 30GB to go

No. Even if that were the case, the bytes written figure would be larger.

[abeconnally]

Overheating or broken storage device? Other than that I'm out of ideas.

I've now tried it with a couple of storage devices, now, and it's the same thing, it'll do about 300K, then bogs down

[abeconnally]

I'm going to compress the folder into a single file and download like that and see if there is an issue