Feature Request: Verify Transfered File Integrity

Moderator: Project members

Message
Author
User avatar
botg
Site Admin
Posts: 33411
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: Feature Request: Verify Transfered File Integrity

#16 Post by botg » 2008-10-17 17:52

Then tell your government to standardize a hash command for FTP. Once the standard is out, I can implement it within minutes.

da chicken
226 Transfer OK
Posts: 619
Joined: 2005-11-02 06:41

Re: Feature Request: Verify Transfered File Integrity

#17 Post by da chicken » 2008-10-19 04:33

I'm not saying there's anything you can do, botg. I'm just saying that sftp and ftps don't meet the needs of cases where data integrity needs to be verified. There's nothing for you to do because there is no standard mechanism for this manner of file transfer. It is assumed the files will be digitally signed at the remote site and then have that signature verified after transfer. The only systems that integrate data signature signing into secure connection file transfer are proprietary at the moment.

I'm just making the point that dismissing the feature request as "FTPS or SFTP is good enough" is not correct. The correct response is simply that there is no standard you can implement so the feature is currently beyond the scope of FileZilla. My understanding is that FileZilla is RFC-only FTP/FTPS/SFTP plus implicit FTPS, and I don't think you want to implement anything other than implicit FTPS that's not in an RFC. It's also just a transfer client. It's not meant for full website management, complete backup support, scripting interfaces, etc. XCRC, XMD5 and XSHA1 are the closest things there are beyond WS_FTP's choice to integrate OpenPGP, which is not that different from manual creation of signatures and manual checking. None of those are standards.

LuarAzul
500 Command not understood
Posts: 2
Joined: 2008-10-17 08:01

Re: Feature Request: Verify Transfered File Integrity

#18 Post by LuarAzul » 2008-12-01 18:34

botg wrote:Just use FTP over TLS (FTPS), it ensures your data stays intact during transfer.
Thanks! I'm really new to all these things and I didn't know I could do that. In any case I have transferred hundreds of gigabytes in the last couple of weeks through FileZilla in plain FTP and I have yet to find a single error (I know because I am using Winrar, which has CRC32 checks). So it ceased to be a top priority for me.

Thanks for the Excellent work!

User avatar
boco
Contributor
Posts: 25490
Joined: 2006-05-01 03:28
Location: Germany

Re: Feature Request: Verify Transfered File Integrity

#19 Post by boco » 2008-12-02 23:57

If everything is configured correctly, and all evil hard- and software and driver issues are out of the way, errors are actually very rare. I must have now transferred several Terabytes and not a single error. 8)
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Do yourself a favor and read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
### END SIGNATURE BLOCK ###

da chicken
226 Transfer OK
Posts: 619
Joined: 2005-11-02 06:41

Re: Feature Request: Verify Transfered File Integrity

#20 Post by da chicken » 2008-12-03 04:40

Actually, here, the CRC32 checks built into the archive file (.zip and .rar files have them) are your verification. Gzip, bzip2 and 7z files also have CRC32 information.

Note that CRC32 verification is generally sufficient to detect inadvertent errors, but it's not sufficient to detect intended flaws. That is, it's not very difficult mathematically to make a second file of equal size with the same CRC32 checks by brute force. MD5 or SHA-1 are recommended, although truthfully you have to go to the SHA-2 family to get a verification algorithm that hasn't been compromised (there are methods faster than brute force for MD5 and SHA-1). Unless you have reason to think you're actively being attacked or need to protect against such attacks, CRC is probably good enough.

twanj
500 Syntax error
Posts: 12
Joined: 2008-01-24 19:10
First name: Ant
Last name: Bryan
Location: Pompano Beach, Florida

Re: Feature Request: Verify Transfered File Integrity

#21 Post by twanj » 2009-10-19 17:22

botg wrote:I need formal specifications from one of the following sources/organizations:
- RFC
- IETF draft
- W3C
- ISO

I won't implement anything else.
Here's the IETF Internet Draft.

User avatar
botg
Site Admin
Posts: 33411
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: Feature Request: Verify Transfered File Integrity

#22 Post by botg » 2009-10-19 21:37

twanj, how is that of any relevance in this context? Did you even read that draft or are you only trolling?

twanj
500 Syntax error
Posts: 12
Joined: 2008-01-24 19:10
First name: Ant
Last name: Bryan
Location: Pompano Beach, Florida

Re: Feature Request: Verify Transfered File Integrity

#23 Post by twanj » 2009-10-20 04:45

botg wrote:twanj, how is that of any relevance in this context? Did you even read that draft or are you only trolling?
My earlier post for why I think it's relevant.

Something like Instance Digests (whole file checksums) or Content-MD5 (partial file checksums) but for FTP, not HTTP would be useful, wouldn't they? Nothing standardized exists yet?

I'm working on expanding the use of Instance Digests within the IETF right now. I don't know much about FTP, if you'd like to collaborate on something similar let me know. It would fit in with the other stuff I'm doing.

User avatar
botg
Site Admin
Posts: 33411
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: Feature Request: Verify Transfered File Integrity

#24 Post by botg » 2009-10-20 08:19

Some FTP specific solution is needed.

twanj
500 Syntax error
Posts: 12
Joined: 2008-01-24 19:10
First name: Ant
Last name: Bryan
Location: Pompano Beach, Florida

Re: Feature Request: Verify Transfered File Integrity

#25 Post by twanj » 2009-10-20 12:02

Ok, I'll do some digging.

What about MIC?

User avatar
botg
Site Admin
Posts: 33411
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: Feature Request: Verify Transfered File Integrity

#26 Post by botg » 2009-10-20 12:11

Depends on the used authentication scheme. And in case of AUTH TLS, MIC isn't even needed as TLS is inherently protected.

twanj
500 Syntax error
Posts: 12
Joined: 2008-01-24 19:10
First name: Ant
Last name: Bryan
Location: Pompano Beach, Florida

Re: Feature Request: Verify Transfered File Integrity

#27 Post by twanj » 2009-10-20 18:16

Seems like there's a need for a checksum feature.

Is this something you would publicly support on mailing lists?

Do you think other application authors would?

How supported are the unofficial features like XMD5, XSHA1 etc?

User avatar
botg
Site Admin
Posts: 33411
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: Feature Request: Verify Transfered File Integrity

#28 Post by botg » 2009-10-20 20:32

Seems like there's a need for a checksum feature.
Yes. If I were to design such a feature I'd add a new command (let's say HASH for the purpose of this discussion) that has two arguments. The first being the name of the hash/checksum algorithm and the second one being a filename. The command should return the hash be in the canonical hex-encoded form.


hash = "HASH" SP <filename> SP <filename>
hash-response = "213" SP 1*HEXDIGIT CRLF

Example:

HASH sha-1 foo.txt
213 69d9e6b36078aa77aa2e8e93f88fdf6ce9dc7782

A server that supports HASH should advertise it in feat response with a list of all supported hash/checksum algorithms, e.g.

FEAT
[...]
HASH crc32,sha-1,md5,sha512
[...]

To verify integrity on download a client would do the following:
1) Get file modification time
2) Get hash
3) Download
4) Get modification time again. If it is different, remote file has changed during transfer. Abort and start over
5) Compare hashes

This is how I would implement such a feature.
Is this something you would publicly support on mailing lists?
What exactly do you mean by that?
How supported are the unofficial features like XMD5, XSHA1 etc?
Not at all since not formerly specified and standardized.

User avatar
botg
Site Admin
Posts: 33411
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: Feature Request: Verify Transfered File Integrity

#29 Post by botg » 2009-10-20 20:41

Actually, forget my proposal, there's something wrong with it. Interacts funnily with the MFMT command.

twanj
500 Syntax error
Posts: 12
Joined: 2008-01-24 19:10
First name: Ant
Last name: Bryan
Location: Pompano Beach, Florida

Re: Feature Request: Verify Transfered File Integrity

#30 Post by twanj » 2009-10-20 21:01

botg wrote:
Seems like there's a need for a checksum feature.
Yes. If I were to design such a feature I'd add a new command (let's say HASH for the purpose of this discussion) that has two arguments. The first being the name of the hash/checksum algorithm and the second one being a filename. The command should return the hash be in the canonical hex-encoded form.


hash = "HASH" SP <filename> SP <filename>
hash-response = "213" SP 1*HEXDIGIT CRLF

Example:

HASH sha-1 foo.txt
213 69d9e6b36078aa77aa2e8e93f88fdf6ce9dc7782

A server that supports HASH should advertise it in feat response with a list of all supported hash/checksum algorithms, e.g.

FEAT
[...]
HASH crc32,sha-1,md5,sha512
[...]

To verify integrity on download a client would do the following:
1) Get file modification time
2) Get hash
3) Download
4) Get modification time again. If it is different, remote file has changed during transfer. Abort and start over
5) Compare hashes

This is how I would implement such a feature.
That's what I had written up, but I was thinking DIG or DGST but HASH is good!
botg wrote:Actually, forget my proposal, there's something wrong with it. Interacts funnily with the MFMT command.
Hmm, is it insurmountable or is there a way around it?
botg wrote:
Is this something you would publicly support on mailing lists?
What exactly do you mean by that?
Would you write on an IETF mailing list that you think this is a needed idea or that you will implement it in an application if it is standardized?
botg wrote:
How supported are the unofficial features like XMD5, XSHA1 etc?


Not at all since not formerly specified and standardized.
But...at least a few apps support those, right? I know it's not universal.

Post Reply