Feature Request: Verify Transfered File Integrity

[botg]

Then tell your government to standardize a hash command for FTP. Once the standard is out, I can implement it within minutes.

[da chicken]

I'm not saying there's anything you can do, botg. I'm just saying that sftp and ftps don't meet the needs of cases where data integrity needs to be verified. There's nothing for you to do because there is no standard mechanism for this manner of file transfer. It is assumed the files will be digitally signed at the remote site and then have that signature verified after transfer. The only systems that integrate data signature signing into secure connection file transfer are proprietary at the moment.

I'm just making the point that dismissing the feature request as "FTPS or SFTP is good enough" is not correct. The correct response is simply that there is no standard you can implement so the feature is currently beyond the scope of FileZilla. My understanding is that FileZilla is RFC-only FTP/FTPS/SFTP plus implicit FTPS, and I don't think you want to implement anything other than implicit FTPS that's not in an RFC. It's also just a transfer client. It's not meant for full website management, complete backup support, scripting interfaces, etc. XCRC, XMD5 and XSHA1 are the closest things there are beyond WS_FTP's choice to integrate OpenPGP, which is not that different from manual creation of signatures and manual checking. None of those are standards.

[LuarAzul]

Just use FTP over TLS (FTPS), it ensures your data stays intact during transfer.

Thanks! I'm really new to all these things and I didn't know I could do that. In any case I have transferred hundreds of gigabytes in the last couple of weeks through FileZilla in plain FTP and I have yet to find a single error (I know because I am using Winrar, which has CRC32 checks). So it ceased to be a top priority for me.

Thanks for the Excellent work!

[boco]

If everything is configured correctly, and all evil hard- and software and driver issues are out of the way, errors are actually very rare. I must have now transferred several Terabytes and not a single error.

[da chicken]

Actually, here, the CRC32 checks built into the archive file (.zip and .rar files have them) are your verification. Gzip, bzip2 and 7z files also have CRC32 information.

Note that CRC32 verification is generally sufficient to detect inadvertent errors, but it's not sufficient to detect intended flaws. That is, it's not very difficult mathematically to make a second file of equal size with the same CRC32 checks by brute force. MD5 or SHA-1 are recommended, although truthfully you have to go to the SHA-2 family to get a verification algorithm that hasn't been compromised (there are methods faster than brute force for MD5 and SHA-1). Unless you have reason to think you're actively being attacked or need to protect against such attacks, CRC is probably good enough.

[twanj]

I need formal specifications from one of the following sources/organizations:
- RFC
- IETF draft
- W3C
- ISO

I won't implement anything else.

Here's the IETF Internet Draft.

[botg]

twanj, how is that of any relevance in this context? Did you even read that draft or are you only trolling?

[twanj]

twanj, how is that of any relevance in this context? Did you even read that draft or are you only trolling?

My earlier post for why I think it's relevant.

Something like Instance Digests (whole file checksums) or Content-MD5 (partial file checksums) but for FTP, not HTTP would be useful, wouldn't they? Nothing standardized exists yet?

I'm working on expanding the use of Instance Digests within the IETF right now. I don't know much about FTP, if you'd like to collaborate on something similar let me know. It would fit in with the other stuff I'm doing.

[botg]

Some FTP specific solution is needed.

[twanj]

Ok, I'll do some digging.

What about MIC?

[botg]

Depends on the used authentication scheme. And in case of AUTH TLS, MIC isn't even needed as TLS is inherently protected.

[twanj]

Seems like there's a need for a checksum feature.

Is this something you would publicly support on mailing lists?

Do you think other application authors would?

How supported are the unofficial features like XMD5, XSHA1 etc?

[botg]

Seems like there's a need for a checksum feature.

Yes. If I were to design such a feature I'd add a new command (let's say HASH for the purpose of this discussion) that has two arguments. The first being the name of the hash/checksum algorithm and the second one being a filename. The command should return the hash be in the canonical hex-encoded form.


hash = "HASH" SP <filename> SP <filename>
hash-response = "213" SP 1*HEXDIGIT CRLF

Example:

HASH sha-1 foo.txt
213 69d9e6b36078aa77aa2e8e93f88fdf6ce9dc7782

A server that supports HASH should advertise it in feat response with a list of all supported hash/checksum algorithms, e.g.

FEAT
[...]
HASH crc32,sha-1,md5,sha512
[...]

To verify integrity on download a client would do the following:
1) Get file modification time
2) Get hash
3) Download
4) Get modification time again. If it is different, remote file has changed during transfer. Abort and start over
5) Compare hashes

This is how I would implement such a feature.

Is this something you would publicly support on mailing lists?

What exactly do you mean by that?

How supported are the unofficial features like XMD5, XSHA1 etc?

Not at all since not formerly specified and standardized.

[botg]

Actually, forget my proposal, there's something wrong with it. Interacts funnily with the MFMT command.

[twanj]

Seems like there's a need for a checksum feature.

Yes. If I were to design such a feature I'd add a new command (let's say HASH for the purpose of this discussion) that has two arguments. The first being the name of the hash/checksum algorithm and the second one being a filename. The command should return the hash be in the canonical hex-encoded form.


hash = "HASH" SP <filename> SP <filename>
hash-response = "213" SP 1*HEXDIGIT CRLF

Example:

HASH sha-1 foo.txt
213 69d9e6b36078aa77aa2e8e93f88fdf6ce9dc7782

A server that supports HASH should advertise it in feat response with a list of all supported hash/checksum algorithms, e.g.

FEAT
[...]
HASH crc32,sha-1,md5,sha512
[...]

To verify integrity on download a client would do the following:
1) Get file modification time
2) Get hash
3) Download
4) Get modification time again. If it is different, remote file has changed during transfer. Abort and start over
5) Compare hashes

This is how I would implement such a feature.

That's what I had written up, but I was thinking DIG or DGST but HASH is good!

Actually, forget my proposal, there's something wrong with it. Interacts funnily with the MFMT command.

Hmm, is it insurmountable or is there a way around it?

Is this something you would publicly support on mailing lists?

What exactly do you mean by that?

Would you write on an IETF mailing list that you think this is a needed idea or that you will implement it in an application if it is standardized?

How supported are the unofficial features like XMD5, XSHA1 etc?

Not at all since not formerly specified and standardized.

But...at least a few apps support those, right? I know it's not universal.