How to keep duplicates with FTP

[ddmeek]

I am trying to download the entire contents of my online box account (1.3 tb) to a hard drive, and am using Firezilla. I can download 779 GB, and then it seems that the download has completed, i.e. I keep re-running the FTP and it doesn't add any new files to the directory. I'm assuming this is because a large number of the files are photos, and many probably have the same auto-generated numeric names. I don't want to overwrite, or skip them, which seem to be the only choices available within Firezilla; does anyone know a way to download all files, including duplicates?

[boco]

If the "duplicates" each are in separate directories, FileZilla will download all of them.

Unfortunately, when the local system is Windows, there's an edge case where you cannot store all files locally as-is. FTP servers are usually on UNIX-like (POSIX) filesystems, those are case sensitive (e. g. Hello.jpg / hello.jpg / hello.JPG are three separate files). However, Windows' filesystems are case insensitive, to Windows these three filenames are the same. So you can only store one of the files in the same directory.

The best solution would be to backup to a local Linux machine (e.g. with ext4 / BTRFS etc. filesystems), then that restriction does not apply.


Hint: Backup to a local Linux VM and then tar the files into a tarball (there's no compression so it is fast). The resulting tarball can be stored on Windows as it's only one file.

[ddmeek]

Thanks for your insight; so the "duplicates" are likely within different folders, but within the same master folder (i.e. Photos). If that's case do you think Firezilla is still downloading them? It's a clear folder where there is a difference: online "Photos" contains approx 630 gb, whereas the FTP'd "Photos" contains 437 gb. Are there other potential issues that might be at play? Thanks in advance,
David

[boco]

Like I said, if they are in different subdirectories, then they are downloaded fine, although preservation of case is NOT guaranteed on a Windows filesystem!

Then, check the file sizes of individual files. The file size of a remote image must be the very same as the local copy, to the Byte. If they aren't, make sure you're not accidentally transfer everything as ASCII Data Type. Images and other binary files get destroyed when being transferred as ASCII.


By the way, simple test:
Set Overwrite action to "Ask".
Start with an empty local directory.
If there then is an Overwrite warning during transfer and there is NO transfer error in the log (as those Overwrite warnings can be also triggered this way), then there's a file with the same name but different case. It will even show you the name(s).