Feature Request: Verify Transfered File Integrity

[khng]

Filezilla is really a great tool

it would be better if it can verify the integrity of transfered file using the command/function support by the ftp server

it would made lots of us more happy

so that we know the file don't broken in the process of transfer

http://www.rhinosoft.com/Knowledgebase/ ... 85&prod=su
http://www.smartftp.com/support/kb/index.php/2549

[botg]

Is there somewhere a formal specification of the XCRC command?

[khng]

Is there somewhere a formal specification of the XCRC command?

Sorry I don't have them

but XCRC is just the CRC32 checksum of file in HEX FORMAT

XMD5 just the MD5 checksum of file in HEX FORMAT

here is copy from Gene6 ftp server manual

XCRC "filename" SP EP
Calculate the CRC32 of filename from SP to EP.
SP − Starting Point in bytes
EP − Ending Point in bytes
Server replies:
¨ 250 [Followed by the checksum]
¨ 501 Bad or missing parameters.
¨ 504 Command not implemented for the specified argument.

XMD5 "filename" SP EP
Calculate the MD5 of filename from SP to EP.
SP − Starting Point in bytes
EP − Ending Point in bytes
Server replies:
¨ 250 [Followed by the checksum]
¨ 501 Bad or missing parameters.
¨ 504 Command not implemented for the specified argumen

note:
both SP & EP is not mandatory

[c0d3h4x0r]

Would you all PLEASE, FOR THE LOVE OF GOD, implement XCRC or some other kind of easy file-integrity checking? Better yet, also offer the ability for a transfer to "only overwrite if hash is different" so that I can easily push known good copies of files around to destinations that contain corrupted copies without having to re-upload EVERYTHING.

It's not like this is a new feature request -- I did a quick search and found the same request in a posting dated last year (2006). It just seems like nothing is getting done about it. And that's a huge shame because FileZilla would be so great if it simply had this feature.

But as it stands, any FTP program is completely useless if you have no idea if the files got transferred correctly. You don't always have a shell account on the destination that you can log into and use to run hash checks manually.

Until such functionality gets implemented in FileZilla, I'll be using SmartFTP instead... which is a shame because I would prefer to use an open-source solution.

[eyebex]

Please note that the whole checksum / hash stuff is not an official standard in the sense that there is no RFC document for it. Different vendors of Windows servers / clients have semi-formal specifications on their sites (which slightly differ), that's all. I guess until there is a real RFC document and these commands get accepted in the Linux / Unix world and implemented in more servers, they probably won't make their way into FileZilla, at least not by the main developers. Feel free to submit a patch, though

[snakyjake]

+1

I just noticed the nice feature on IPswitch's WS_FTP Pro and on SmartFTP that check the transfer integrity using checksums.

With FileZilla I have to do the checksums via MD5 manually. I create a MD5 file on the client, then I transfer the MD5 to the server. Then the recipient of a file manually checks against the uploaded MD5. Easy to do manually with just a few files. But I'm transferring many, many files. I use FTP for my monthly backups. And it is highly desirable to know my backup matches the source.

Thanks,

Jake

[richbeales]

even if it just detected whether the local file and remote file had the same number of bytes, it would be very helpful.

If a timeout occurs then the file will be half copied, yet still count as a succesful transfer, which is wrong.

Are there any plans to implement this?

[boco]

even if it just detected whether the local file and remote file had the same number of bytes, it would be very helpful.

If a timeout occurs then the file will be half copied, yet still count as a succesful transfer, which is wrong.

Are there any plans to implement this?

Won't work with ASCII mode because of possible line ending conversion.

[twanj]

I'd like to see this too.

Besides XMD5, you can also get the checksum from a metalink, which is an XML file that describes downloads with info like URLs & checksums. (I posted a feature request here and if you think the feature fits, I'd post a bounty). I understand that this is not a download manager, since things like FTP clients & web browsers support metalink, it's more of a general download format. Some apps just use it to add a bunch of files to a transfer queue. Some P2P apps let you use it to describe your library & share a set of specific files.

Since you control the server & client, you could do some interesting things...when uploading, you could use the metalink to describe whole file or partial file checksums, then the server could verify that there were no errors. If there were a few errors, the server could scan the file & request just the chunks that had errors. That could be really nice for large files.

Here's an example of a single file .metalink:

Code: Select all

 <metalink version="3.0" xmlns="http://metalinker.org">
   <files>
     <file name="example.ext">
     <verification>
       <hash type="md5">example-md5-hash</hash>
       <hash type="sha1">example-sha1-hash</hash>
     </verification>
     <resources>
       <url type="ftp" location="us" preference="90">ftp://ftp.example.com/example.ext</url>
       <url type="ftp" location="uk" preference="90">ftp://ftp.example.net/example.ext</url>
       <url type="http" location="us" preference="90">http://example.com/example.ext</url> 
       <url type="http" location="de" preference="90">http://example.net/example.ext</url>
     </resources>
     </file>
   </files>
 </metalink>

[botg]

I need formal specifications from one of the following sources/organizations:
- RFC
- IETF draft
- W3C
- ISO

I won't implement anything else.

[da chicken]

XCRC, XSHA1, and XMD5 are all proprietary file integrity commands, although a lot of different servers support them:
http://www.smartftp.com/support/kb/2549
http://help.globalscape.com/help/secure ... ecking.htm

It is not defined by any standards organization that I'm aware of.

I really have to question the real use of such a thing, as it would seriously load the server's CPU to do this on any extended basis for more than one user simultaneously. It's not like we're using UDP/TFTP here so the packets already have a TCP/IP checksum, meaning that you'd have to have bit errors at the OS or disk level to have a problem. If you're that paranoid and must do this, you should already be creating a separate MD5 or SFV file, putting the file in a ZIP archive (which has it's own CRC checksum even if you don't compress it), or generating PAR2 parity files to send with your data.

[boco]

I found a draft describing the MFF (Modify Facts) command. It's the complimentary to the MLSx commands.

[LuarAzul]

I need formal specifications from one of the following sources/organizations:
- RFC
- IETF draft
- W3C
- ISO

I won't implement anything else.

I have four pcs connected through a router and I used to transfer files between them through the "file and printer sharing for microsoft networks". However the speed was simply awful. I don't understand why but the most I got was about 13% of bandwidth use, and sometimes it crawled to 1%. I tried to solve or at least identify the problem with no success (disabling firewalls and antivirus did nothing, and drivers were not the problem), but a couple of days ago I remembered I could just use an ftp server to do the trick. The thing works wonders: now I have a bandwidth usage continually above 90% which is great, and between all my pcs. However I'd like to be sure that the files are the same. Compressed rar files present no problem but with any other files a doubt always remains. I've been manually checking the CRC32 checksums in some very large files and so far everything is fine. But I just don't understand the need for this RFC thing. I mean eventually it will came, but in the meantime we could just implement the XCRC like other FTP servers and clients already do, and we would modify it in the future if needed by an upcoming RFC. In any case future development may always imply improvements and modifications in the software.

I don't understand why is having a "formal" specification so important to the point of disabling an obviously important and (I guess) easy to implement feature??

Thanks.

[botg]

Just use FTP over TLS (FTPS), it ensures your data stays intact during transfer.

[da chicken]

No. That's not the problem that checksums seek to correct. Or rather, it's not the only problem they seek to correct. I've done more research since my previous posts.

TLS and TCP checksums ensure accurate data transfer from the Network and Transport layers of the OSI model. That says nothing about whether the data is intact before it is read, or if it accurate after I write it to disk. If, for example, I'm operating a service to transfer healthcare information in the US, then I need to ensure that a) the data I store is accurate, b) the data I send is accurate, and c) the data I receive is what I was sent and that source can verify it's accuracy. In this case, having data + size + checksum available using the SHA-2 family then I can ascertain that the data is what was sent. This is the entire point of digital signing, and, indeed, some FTP servers and clients (WS_FTP) implement digital signing requests.

Packet checksums are the equivalent of a wax seal on an envelope. Yes, the data will not be damaged or tampered with in transit, but transit is not the sole cause for loss of data integrity. MySQL developers infamously tried to get away with the same kind of thing before it was ACID-compliant. They said you could do everything that ACID compliance requires within the software application. DBAs with any real experience or training knew that to be a load of horseshit, and it's the same reason MySQL got flak for having fsync turned off by default.

It's arguable whether XCRC, XMD5, or XSHA1 are good enough for anything, though, since all three algorithms have been compromised. It's also arguable whether FTP, FTPS, or SFTP are responsible for this kind of digitally signed data transfer, or if there ought to be an even more secure protocol for the exchange of data whose integrity must be guaranteed beyond the scope of mere network transit. Leaving it up to the OS and assuming it does things right is exactly what you can't do.