Feature Request: Verify Transfered File Integrity

Moderator: Project members

Message
Author
User avatar
botg
Site Admin
Posts: 33419
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: Feature Request: Verify Transfered File Integrity

#31 Post by botg » 2009-10-20 21:52

botg wrote:Actually, forget my proposal, there's something wrong with it. Interacts funnily with the MFMT command.
Hmm, is it insurmountable or is there a way around it?
There is a way around it. Have HASH return a timestamp as well as the HASH. And another command that checks if the server has executed any command that could have changed the hash. Let's call it CHSH (changed hash) that takes that timestamp as well as a filename and returns either "true", "false" or "unknown" (quoted).

Example 1: using two FTP connections:
1> HASH foo.txt
<1 213 20091020214634 69d9e6b36078aa77aa2e8e93f88fdf6ce9dc7782
1> CHSH 20091020214634 foo.txt
<1 Validity of hash calculated at 20091020214634 is "true"

Example 2 using two FTP connections:
1> HASH foo.txt
<1 213 20091020214634 69d9e6b36078aa77aa2e8e93f88fdf6ce9dc7782
2> STOR foo.txt (simplified command sequence for the purpose of example)
<2 200 OK
2> MFMT 20091020214634 foo.txt
<2 200 OK
1> CHSH 20091020214634 foo.txt
<1 Validity of hash calculated at 20091020214634 is "false"

Example 3:
1> CHSH 198091020214634 justsomerandomfilename.txt
<1 Validity of hash calculated at 198091020214634 is "unknown"

A server should at least track the changes made to the last file HASH was called on, for each connection. If resources permit, server may keep track of modifications for a longer time. Likewise, servers may cache hashes to improve performance if they keep track of changes to those files.



Now, those specifications aside, I can't but ask myself: Why? There's already FTP over TLS. Files are inherently protected from being corrupted during transfer, with the exception being bugs in either the server or the client. A hash command wouldn't really change this, bugs could still cause it to report success and matching hashes when in fact the file hasn't been written correctly.

twanj
500 Syntax error
Posts: 12
Joined: 2008-01-24 19:10
First name: Ant
Last name: Bryan
Location: Pompano Beach, Florida

Re: Feature Request: Verify Transfered File Integrity

#32 Post by twanj » 2009-10-21 21:23

botg wrote: Now, those specifications aside, I can't but ask myself: Why? There's already FTP over TLS. Files are inherently protected from being corrupted during transfer, with the exception being bugs in either the server or the client. A hash command wouldn't really change this, bugs could still cause it to report success and matching hashes when in fact the file hasn't been written correctly.
I'm interested in it for early file mismatch detection. If you're interested in a certain file, and that doesn't match then you don't download it.

We use ETags and Instance Digests with HTTP but there isn't anything for FTP that I know of.

twanj
500 Syntax error
Posts: 12
Joined: 2008-01-24 19:10
First name: Ant
Last name: Bryan
Location: Pompano Beach, Florida

Re: Feature Request: Verify Transfered File Integrity

#33 Post by twanj » 2009-10-22 00:14

A partial list of software supporting some non-standard XMD5/XSHA1/XCRC etc commands

SmartFTP client
http://www.smartftp.com/features/
ProFTPD module mod_digest
http://www.smartftp.com/oss/proftpd/mod_digest.html
FireFTP
http://fireftp.mozdev.org/features.html
JAFS
http://www.sbbi.net/site/jafs/features.html
Gene6 FTP Server
http://www.g6ftpserver.com/en/information#features
WS_FTP client / server / MOVEit DMZ
http://ipswitchft.custhelp.com/app/answ ... /166/sno/1
Akamai NetStorage p17-18
http://pigdogslow.dyndns.org/NetStorage_UserGuide.pdf
Null FTP
http://www.sharewareconnection.com/null ... nt-pro.htm
IceWarp FTP Server
http://www.icewarp.com/products/ftp_server/
FileCOPA FTP Server
http://www.filecopa-example.com/features.html
Cerbeus FTP server
http://www.softpedia.com/progChangelog/ ... -1904.html
Starksoft Ftp Component for .NET / Mono
http://www.starksoft.com/prod_ftp.html
Nofeel FTP server
http://www.nftpserver.com/history.php
zFTPServer
GoldenGate FTP (Ftp Full Java Server)
RaidenFTPD32 FTP server
Last edited by twanj on 2009-10-22 12:58, edited 1 time in total.

User avatar
botg
Site Admin
Posts: 33419
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: Feature Request: Verify Transfered File Integrity

#34 Post by botg » 2009-10-22 09:19

Make it "some non-standard XMD5/XSHA1/XCRC etc commands", there could easily be a dozen different commands with the same name.

Lorne
500 Command not understood
Posts: 1
Joined: 2010-01-13 16:31
First name: Lorne
Last name: Liechty

Re: Feature Request: Verify Transfered File Integrity

#35 Post by Lorne » 2010-01-13 16:41

I know that this is a long and possibly dead thread, but I'd like to throw in my two-cents.

I send lots of data back and forth between the US and India and I've had many problems with file corruption even over FTPS. I'm not all that familiar with how each of these protocols work, but I'm guessing that the files get corrupted when the internet connection is unstable (happens fairly frequently).

Since a single file that I send is usually around 300MB, it has gotten to the point that almost every transfer gets corrupted. To mitigate this, I've started zipping all the files into a single, split archive with 50MB sized parts. Then I create a file with a list of the MD5 checksums of all the parts. After the transfer is complete my colleagues in India check the files and then, if necessary, request a resend of any corrupted files (usually one or two are corrupted).

So I wouldn't say that FTPS is perfect for verifying file integrity. I'd love to see this file-transfer-verification feature implemented in Filezilla.

eyebex
226 Transfer OK
Posts: 171
Joined: 2004-04-02 15:24

Re: Feature Request: Verify Transfered File Integrity

#36 Post by eyebex » 2010-01-18 07:48

Lorne, a probably better method than resending the broken parts would be to use a tool like [1]. It creates partity sums, very similar to how RAID-5 works, which can be used to repair the transferred data without transferring it again.

[1] http://www.quickpar.org.uk/

User avatar
boco
Contributor
Posts: 25494
Joined: 2006-05-01 03:28
Location: Germany

Re: Feature Request: Verify Transfered File Integrity

#37 Post by boco » 2010-01-18 09:10

I'm using QuickPAR, too, works very well.
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Do yourself a favor and read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
### END SIGNATURE BLOCK ###

twanj
500 Syntax error
Posts: 12
Joined: 2008-01-24 19:10
First name: Ant
Last name: Bryan
Location: Pompano Beach, Florida

Re: Feature Request: Verify Transfered File Integrity

#38 Post by twanj » 2010-03-26 21:32

for those interested in this feature, here's a first pass at an internet draft

http://tools.ietf.org/html/draft-bryan-ftp-hash

User avatar
boco
Contributor
Posts: 25494
Joined: 2006-05-01 03:28
Location: Germany

Re: Feature Request: Verify Transfered File Integrity

#39 Post by boco » 2010-03-27 08:07

275 Authors' Addresses
.
.
.
284 Tim Kosse
:P :P :P :mrgreen:
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Do yourself a favor and read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
### END SIGNATURE BLOCK ###

camillo777
504 Command not implemented
Posts: 8
Joined: 2009-12-21 11:15
First name: camillo
Last name: camillo

Re: Feature Request: Verify Transfered File Integrity

#40 Post by camillo777 » 2010-05-18 12:50

Any update on this?
Such a feature would be appreciated and could boost up the use of both filezilla server and client.

Camillo

twanj
500 Syntax error
Posts: 12
Joined: 2008-01-24 19:10
First name: Ant
Last name: Bryan
Location: Pompano Beach, Florida

Re: Feature Request: Verify Transfered File Integrity

#41 Post by twanj » 2010-05-18 16:27

camillo777 wrote:Any update on this?
Such a feature would be appreciated and could boost up the use of both filezilla server and client.
still in progress. these things take a while :)

twanj
500 Syntax error
Posts: 12
Joined: 2008-01-24 19:10
First name: Ant
Last name: Bryan
Location: Pompano Beach, Florida

Re: Feature Request: Verify Transfered File Integrity

#42 Post by twanj » 2010-06-13 20:29

there is a new BOF meeting about FTP extensions at the next IETF meeting: http://trac.tools.ietf.org/bof/trac/att ... tp-bof.txt

it was brought on by this draft and http://tools.ietf.org/html/draft-hethmo ... -ftp-hosts :)

there are a number of IDs that did not finish to RFC status that need to be reviewed, to determine if they should be revived or left abandoned.

Macktek
550 File not found
Posts: 33
Joined: 2016-03-03 19:04
First name: Mack

Re: Feature Request: Verify Transfered File Integrity

#43 Post by Macktek » 2016-03-03 20:03

I have been using Filezilla for years.
However, I have encountered corrupt transfers over the last few years.
Its not really a problem with Filezilla, but just the overall process, where corruption can occur anywhere in the chain.
What is needed is some way to verify file integrity after the transfer is complete.

The overall reality of this problem is that despite a desperate call for checksums, filezilla has declined to implement anything.

I do understand that it may not be in some "standard"... nevertheless it is needed.

Waiting is just causing everyone problems.

Is it not reasonable to create an optional extension that implements a checksum or some other method to verify file integrity?

If not... then I really have no choice than to abandon filezilla for an FTP server and client that has this functionality.
I am really preferring not to do this, but in the last few days I have had to spend hours tracking down bad files that "transferred" according to filezilla.
Yet, ultimately, were corrupt.
This is a real problem. We need some kind of checksum / verification system.

So please... add it in even if its an optional extension.
If I have missed this as an option somewhere in Filezilla, I apologize.... If it is there how do I turn it on?

Thank you for your consideration.
Macktek, LLC

User avatar
botg
Site Admin
Posts: 33419
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: Feature Request: Verify Transfered File Integrity

#44 Post by botg » 2016-03-03 20:21

FTP over TLS detects corruption in transit. Checksummed filesystems such as btrfs detect on-disk corruption. The missing link between the two is the used FTP software, but if that's causing the corruption, it probably doesn't implement the checking correctly either.

Macktek
550 File not found
Posts: 33
Joined: 2016-03-03 19:04
First name: Mack

Re: Feature Request: Verify Transfered File Integrity

#45 Post by Macktek » 2016-03-04 00:39

You were once championing the idea of checksums here: https://tools.ietf.org/html/draft-bryan-ftpext-hash-02
Network Working Group A. Bryan
Internet-Draft T. Kosse
Intended status: Standards Track D. Stenberg
Expires: August 22, 2013 February 18, 2013


File Transfer Protocol HASH Command for Cryptographic Hashes
draft-bryan-ftpext-hash-02

Abstract

The File Transfer Protocol does not offer any method to verify the
integrity of a transferred file, nor can two files be compared
against each other without actually transferring them first.
...

If the groups creating the standards are not willing to do the right thing...
then why not do it yourself? The likelihood is they sit on it for near eternity or follow your example anyway. (especially given your expertise)

Post Reply