Feature Request: Verify Transfered File Integrity

[botg]

The checksum feature is no longer needed to ensure the integrity of transferred files. FTP over TLS takes care of the integrity protection, with the added benefit of encryption and authentication of the data connection against the control connection.

If you are worried about data on disk becoming corrupted, use a check-summed filesystem such as Linux' btrfs or Windows' ReFS, both of which can even repair your data if it is stored redundantly.

[Macktek]

Thanks.

I have implemented FTP over TLS. I guess it will take some time to see if the errors are decreased.

EDIT:
I have encountered an issue of file corruption despite the usage of FTP over TLS when the connection is interrupted.
This is more easily tested with large files so that is what I used.

Transfer LargeFile.doc 100,000KB from client to Server (done).
Changed LargeFile.doc on client slightly.
Initiate Transfer LargeFile.doc 100,000KB from client to Server and disconnect from LAN in middle of transfer.

This results in a 0 byte file on the server. Which is not acceptable because it destroys a good (albeit older) copy of the file on the server before it receives an updated version of the entire file from the client.
In my case, It is acceptable to have an older version stay on the server, versus keep a corrupted zero byte file.

In Windows, the OS protects files from incomplete copy transfer but I am not seeing this with Filezilla on Windows.

Is there a way to tell Filezilla to perform a safe copy over the existing file only when the file is completely received? ( I am aware this may add some overhead while windows renames the copy and deletes the original)

Thank you for any assistance.

[botg]

I have encountered an issue of file corruption despite the usage of FTP over TLS when the connection is interrupted.
This is more easily tested with large files so that is what I used.

Transfer LargeFile.doc 100,000KB from client to Server (done).
Changed LargeFile.doc on client slightly.
Initiate Transfer LargeFile.doc 100,000KB from client to Server and disconnect from LAN in middle of transfer.

This results in a 0 byte file on the server. Which is not acceptable because it destroys a good (albeit older) copy of the file on the server before it receives an updated version of the entire file from the client.

Note that the checksum feature would not have helped in this case either.

Also, both the client and the server would notice the transfer failure and could take appropriate action. FileZilla for example resumes transfers after connection failures.

In Windows, the OS protects files from incomplete copy transfer but I am not seeing this with Filezilla on Windows.

No, it doesn't. Windows merely doesn't update the displayed filesize fast enough.

Is there a way to tell Filezilla to perform a safe copy over the existing file only when the file is completely received? ( I am aware this may add some overhead while windows renames the copy and deletes the original)

Unfortunately no, though I consider adding such a feature.

It may sound simple, but it's actually a very difficult task.

Consider these examples which are made under the assumption that files can be uploaded and resumed at an arbitrary offset:

1) A server might now allow renaming files if the target already exists. So the solution would seem to delete the target before the rename. Turns out deletion is not allowed. Lots of useless temporary files that can accumulate on the server this way, requiring support from the server administrator.

2) Choosing a temporary filename is difficult.

2a) There is the STOU command to generate a unique filename but it cannot be used. The FTP specifications, while requiring that the STOU reply contains the created filename, do not say how precisely the reply has to look like. Guesswork is a really bad idea in this context.

2b) Let's say you have readme.txt, can you use readme.txt.tmp as temporary filename? What if it already exists? What if it exists but the server doesn't show it to the client? What if readme.txt.tmp exceeds the maximum allowed pathname length?

3) What if the target file is a symbolic link or a hardlink? By renaming/deleting you'd destroy the link without affecting the link target

[Macktek]

hmm,

You claim that FTP over TLS obviates checksum, so lets agree on that.

ReFS does not support hard links... so that can't help either.

Re:

1) A server might now allow renaming files if the target already exists. So the solution would seem to delete the target before the rename. Turns out deletion is not allowed. Lots of useless temporary files that can accumulate on the server this way, requiring support from the server administrator

Assuming the server allows renaming of files, then yes, the solution would be to simply name the incoming file some random hash generated name perhaps starting with _eraseme_+the random hash.
I have actually seen this behavior of random temp file names (minus the _eraseme_prefix) while uploading to rented servers in datacenters, so I am pretty sure this exists.

Deletion is not allowed by the "client", but couldn't the server admin allow the server to delete server created files for the purpose of garbage collection?

Obviously, maximum length of the hash could have some limit, so is it really that much of a problem?
(I mean really, the same argument could be applied to anyone connecting from a server that has very long names
Linux has a maximum filename length of 255 characters for most filesystems (including EXT4), and a maximum path of 4096 characters and windows paths are limited to 260 (AFAIK) so does that mean we can't allow clients to connect to windows servers just because of that? Sure its a risk... but.. we still allow it.

It seems to me that much of these limits are "self-imposed".
You could create the hash, truncate the name if it were too long... store the path/name in temp... Delete the fie with the hash name once the copy was completed. (or some variant of that).

Yea, there would still be issues such as write caching... but you can't control those things unless the admin fixes it.
Nevertheless its far better than not having any safety measures.

In answer to your question: what if the file with the random hash already exists... in that case, you simply regen the hash.

[botg]

I'm looking for features that either always work or, if they cannot work, fail reliably. This feature on the other hand would probably work sometimes and fail at other times depending on things like filenames, permissions and current location on the server in a very intransparent way that is hard to convery to users

It seems to me that much of these limits are "self-imposed".

Sadly no. With the advent of the IoT and the limited capabilities of these devices, I fully expect to encounter a large number of highly limited servers in the near future.

In answer to your question: what if the file with the random hash already exists... in that case, you simply regen the hash.

That assumes it is possible to check whether a file exists. Which isn't always the case.

[Macktek]

hmm , why wouldn't there be a "file exists" function? The functionality exists in the form of the Search Function that FZ has , although perhaps its not formalized.
Certainly, if you can perform a simple search function that displays files in a directory, then couldn't you implement a search for the hash file in the current directory and determine if there was a true result? (Since FZ client can do a search for files already).

In regards to less capable servers: Well, I don't agree that FileZilla should "only" cater to the lowest common denominator of functionality in servers.
If a server can't implement a certain function then they can't use it... should not mean that other more capable servers be denied that option.
The admin can decide if the feature can be enabled.


There are already several examples where FZ fails regularly with regard to certain operations when the admin decides to implement or "uncheck" certain features.
Specifically, I refer to: Write privileges.
If the server admin chooses to remove write privs for a specific user, and the client attempt to write... it just fails to write.
The client has very little idea what is happening or why. Yet, the feature to allow /or not allow write access still exists.
So, the logic seems sound to me... that FZ does not "require" a certain feature to always work reliably 100%.
Ie, a file transfer will fail if write privs are turned off.
And if that is the case, why not implement the feature since its limitations would be also be under the control of the admin if they wanted to turn off a feature (supported or not).

[botg]

hmm , why wouldn't there be a "file exists" function? The functionality exists in the form of the Search Function that FZ has , although perhaps its not formalized.

It only shows what the server willingly displays. It does not show what the server hides but still may be there.

There are already several examples where FZ fails regularly with regard to certain operations when the admin decides to implement or "uncheck" certain features.

The difference is in the outcome of the failure. In those other cases, nothing bad happens on failure.

[Macktek]

In the case of not showing hidden files... why would this make a difference?

If someone "guessed" at a hidden file name, and transferred from client to server, would the server actually change the file?

[boco]

It depends on the server configuration, but it is very well possible.

[botg]

What could happen is that, when user tells FileZilla to transfer file A to the server, that file B on the server instead gets overwritten and then renamed to A. In essence it would delete a completely unrelated file. The workaround would be to use long random filenames, but then you run into the issue with length limitations.

[Macktek]

It depends on the server configuration, but it is very well possible.

That would seem like a pretty big hole in security.
It would seem if someone had "hidden files" in an FTP directory on their server to begin with, they ought to know they would be hidden, and at the very least set the appropriate protection bits. What's to stop an entire directory from being deleted?

I don't quite understand why someone would have hidden files on an FTP directory without setting those attributes.
That should fall within the realm of responsibility of the server admin.

Anyway... all I see so far is edge case reasons why this should not be done. And these edge cases are all rare, and very much depending on "improperly" configured servers.

The argument that: very rarely you could possible rename the wrong file or delete one, does not stand well against the fact that you will "ALWAYS" lose your file if the connection is dropped or fails.

[botg]

Another edge-case, even on a perfect server: TOCTTOU.

[Macktek]

Hmm, TOCTTOU is really a higher risk issue to me. As it would be an issue for ALL read/writes not just renaming... for practically every server and would require several levels of protection to prevent, such as limiting access to 1 user until all activity is completed (and that still would not protect it from OS side r/w). AND FZ allows multiple user access to the same folders... despite the very high chance that this can occur... so again, I say... these are not valid reasons to deny implementation.

BTW: The odds of possibly renaming the wrong file name are "astronomically" small if Filezilla requires that the admin set aside a specific area for temp. files. The path of the temp directory could be set in the settings and be write protected from all users except FZ Server.

[botg]

BTW: The odds of possibly renaming the wrong file name are "astronomically" small if Filezilla requires that the admin set aside a specific area for temp. files.

Server administrators would never do that. It's like selling a car that requires new roads to be built specifically for that model of car. Nobody would build the roads.

[Macktek]

The temp folder could also be created with a randomized name and deleted once the transfer is complete. The "admin" could probably care less about it since it would only be holding a few files of short lived duration.
hehe ok ok have it your way. Don't implement it.