Is automatic uploading of changed files still wontfix?

Come here to discuss FileZilla and FTP in general

Moderator: Project members

Message
Author
jumpfroggy
503 Bad sequence of commands
Posts: 18
Joined: 2007-08-25 16:14

Is automatic uploading of changed files still wontfix?

#1 Post by jumpfroggy » 2010-01-22 18:59

First off, thanks so much for filezilla. It's such a great FTP program, and with the add of synced browsing it's gotten easier than I imagined it could. I love using the View/Edit local file to edit a file, and then have FZ ask me when the file's changed if I want to upload it. That's really convenient for working on remote servers.

I would love to be able to have FZ automatically upload the files after they've been changed w/o a prompt. Now, I realize this can result in corrupt data. I also realize it's already been asked a ton before. I just want to ask again - is it still something that wont be supported, or has the philosophy changed and could it be supported in the future?

I know this is an advanced option, but it'd be really amazing to have this work. In order to avoid data corruption, FZ could store the hash of a file, then upload it, then check the hash again. That way if the local file has been modified while uploading, FZ can re-upload the file. So there may be a corrupt version of the file on the server for a brief period, but it would be reliably corrected right after. For the kind of files & situations you'd want to use automatic uploading, this would not be an issue. If it were, you wouldn't want to use automatic uploading in the first place.

Obviously this would be optional and turned off by default. I'd suggest that it would be in the "File has changed" dialog, next to the "Finish editing and delete file" checkbox. I would name it "Remember this decision". Pressing yes would bring up a warning about data corruption, with an option to cancel or ok, and also a "don't show this again" checkbox. This would warn naive users of the dangers, while still allowing power users to use this.

Yes, this is an advanced/risky option, but very convenient. I know botg was against this before, but I don't know if those feelings have changed since.

If none of this is acceptable: does anyone have a recommendation for a good/free/simple app that will just do this kind of automatic updating? I really want to do this with FZ, but if I have to use another program I'd love some recommendations.

Image
Image

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: Is automatic uploading of changed files still wontfix?

#2 Post by botg » 2010-01-22 23:12

Is automatic uploading of changed files still wontfix?
Are users still not searching the forums? It is a technical impossibility, I've explained the reasons a dozen times.

jumpfroggy
503 Bad sequence of commands
Posts: 18
Joined: 2007-08-25 16:14

Re: Is automatic uploading of changed files still wontfix?

#3 Post by jumpfroggy » 2010-01-22 23:59

botg wrote:
Is automatic uploading of changed files still wontfix?
Are users still not searching the forums? It is a technical impossibility, I've explained the reasons a dozen times.
I'm not sure if you've read my full post. It does seem technically possible.

At least that's where we ended up the last time we discussed this, which was a long time ago. I was asking whether the feelings had changed, or if was still the official line.

I understand your reasons for not wanting to do this was due to data corruption, but for the use case I had in mind that would be a non issue. So I was just curious where things stood.

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: Is automatic uploading of changed files still wontfix?

#4 Post by botg » 2010-01-23 08:27

My feelings are of no concern. My opinion however has not changed.

ElPazzo
500 Command not understood
Posts: 1
Joined: 2010-01-24 11:02
First name: Pascal
Last name: Zotto

Re: Is automatic uploading of changed files still wontfix?

#5 Post by ElPazzo » 2010-01-24 12:24

Sad, wanted to ask if such a feature exists or is planned for FZ as I like FZ very much since I tested it. :cry:

jumpfroggy have a look at ftp-uploader:

http://www.ftp-uploader.de/index.php they also offer a very nice freeware editor supporting all programming languages (phase5)

it's in German and I don't know if it exists in other languages but it does exactly what you want and via the scheduler you can also tell it to check the files (it checks the directory you set up for changed files and uploads them) every x seconds/minutes/hours and to upload them. Setting up various tasks you can then even tell it to upload the same files to multiple servers at once. Every file gets a changed and a last uploaded timestamp if change time is newer than upload time then it's uploaded. Though from time to time it resets the upload timestamps completely and everything is uploaded again.

For uploading (if it found anything to upload) it connects to the server automatically and closes connection afterwards.

I use that one a lot when programming on my online shop which has the same source code for all languages but every language is on it's own server. So with this I only need to update every file once and the uploader does the rest.

@ botg: maybe u could get a hint on how to programme such a feature while having a look at ftp-uploader. Dreamweaver also has the automatic upload feature but is quite expensive to get legally and can only upload to one server.
I know it from my own programming experience, quite often I think something is impossible to do but then after some time and cross checking with existing similar features I have the breakthrough for that matter and managed to set it up after lots of testing around.

anyway, very nice programme you made. I like it more than any other ftp programme I tried, even if I miss the feature mentionned above. :wink:

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: Is automatic uploading of changed files still wontfix?

#6 Post by botg » 2010-01-24 12:58

What other programs do is to gamble with your data. There is a very real risk involved and eventually you'll loose your data.

jumpfroggy
503 Bad sequence of commands
Posts: 18
Joined: 2007-08-25 16:14

Re: Is automatic uploading of changed files still wontfix?

#7 Post by jumpfroggy » 2010-01-26 02:20

ElPazzo wrote:Sad, wanted to ask if such a feature exists or is planned for FZ as I like FZ very much since I tested it. :cry:
jumpfroggy have a look at ftp-uploader:
http://www.ftp-uploader.de/index.php they also offer a very nice freeware editor supporting all programming languages (phase5)
ElPazzo: Thanks for the link, I ended up creating my own program. I'll post a link here later when it's in a workable form. It's just a simple python command-line program that watches a folder for any file-modification-time changes, and then uploads any changed files to the corresponding folder on the server via SFTP. Works great for my purposes, and it checks file-modification-time and checksum to make sure it doesn't leave corrupted versions of the file on the remote folder. Very convenient for remote dev when SSHFS or NFS aren't an option.

Ray Yates
500 Command not understood
Posts: 1
Joined: 2010-02-20 14:48
First name: Ray
Last name: Yates

Re: Is automatic uploading of changed files still wontfix?

#8 Post by Ray Yates » 2010-02-20 15:11

bogt said. What other programs do is to gamble with your data. There is a very real risk involved and eventually you'll loose your data.
Let me explain my process. I write web apps. As I work I have my program files open in a text editor.
I compile,
click the FX tab on the task bar,
hit enter,
tab to the browser and refresh window,
tab back to my editor
repeat.

It only takes seconds, but that's not the point.

The only reason I'm on the forum today is to find a way to eliminate hit enter,

With all respect for your achievement, I could care less about occasional data corruption. Really I just don't care. When this happens I would get the occasional Server 500 internal error, repeat the process and it's gone. It's so rare and so easy to overcome, that for me, it's not a problem.

I'm probably not going to convince botg but I'm requesting this feature as well.

My suggestion: Bury a checkbox it in some obscure settings location, provide lots of warnings on a popup why it’s a bad idea, "Use at your own risk!" and then the only question you will get on the forum is "Where is it?"

Respectfully,

George-FZ
500 Command not understood
Posts: 1
Joined: 2010-02-25 09:01

Re: Is automatic uploading of changed files still wontfix?

#9 Post by George-FZ » 2010-02-25 09:15

There was a great program for automatic updates, called "The Homepage Upper". Unfortunately this program won't work on Windows 7. It is surprising but it seems that there is no real alternative.

The Homepage Upper did not care about remote directories, it just monitored the local directory for changed files. When it found new files, it created a list, where one was able to remove files that should not be uploaded. Finally, when everything was OK, an "upload" button was starting the updating process. Of course, there were also filters of files or folders that should never be included in updates, and directory structures always maintained.

I don't know why the FileZilla team finds this functionality 'impossible'. Even if it finds it just unsafe, why not warn for this, and let the user take the risk?

jumpfroggy
503 Bad sequence of commands
Posts: 18
Joined: 2007-08-25 16:14

Re: Is automatic uploading of changed files still wontfix?

#10 Post by jumpfroggy » 2010-02-25 16:01

George-FZ wrote:I don't know why the FileZilla team finds this functionality 'impossible'. Even if it finds it just unsafe, why not warn for this, and let the user take the risk?
Careful here - I don't think the team has ever said that this is impossible, only that they wont do it.

And while I truly wish this were integrated, I think they've made it clear they simply wont change their mind about this. I'll try to cleanup & release my own CLI program for this - works wonders for me, and it's actually quite simple to use.

User avatar
boco
Contributor
Posts: 26899
Joined: 2006-05-01 03:28
Location: Germany

Re: Is automatic uploading of changed files still wontfix?

#11 Post by boco » 2010-02-25 19:33

I don't think the team has ever said that this is impossible, only that they wont do it.
You didn't even read this thread fully.

Quote from botg, 1st reply:
It is a technical impossibility
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Please do yourself a favor and read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
### END SIGNATURE BLOCK ###

jumpfroggy
503 Bad sequence of commands
Posts: 18
Joined: 2007-08-25 16:14

Re: Is automatic uploading of changed files still wontfix?

#12 Post by jumpfroggy » 2010-02-25 19:38

Ok, as a stopgap I've uploaded my (incredibly simple) FTP/SSH auto-sync program here:

https://code.google.com/p/remotefoldersync/

It's very simple. Needs python & paramiko. Just run with the command line args you want, and it'll automatically upload files when they've changed on the local folder. Note - this does not truly "sync" folders - if the two are out of date, the program doesn't know. It simply uploads any files that are changed while the program is running.

So run the program, then modify files on your local folder and see them show up on the remote folder. Very convenient for editing/developing files on a server.

Also note that the uploading makes an accurate copy of what's on your local folder - that means that if a program takes a while to write to the local file, then the remote folder will be an accurate copy of that... which means the remote file will be partially written to. This is alleviated by 1) the program scans every second for changes, 2) it also checks for constantly-written-to files, so it should pick up writes that follow right after, and 3) you are a developer. You know the risks. You're ok with them. The files are non-critical, you wont crash a server with them. You are a grownup (or have the mind of one).

Feedback is welcome. This is not any kind of competition to filezilla - just a nice complement for the lacking feature until they a) decide to add this feature in to appease us, or b) the universe collapses.

jumpfroggy
503 Bad sequence of commands
Posts: 18
Joined: 2007-08-25 16:14

Re: Is automatic uploading of changed files still wontfix?

#13 Post by jumpfroggy » 2010-02-25 19:39

boco wrote:
I don't think the team has ever said that this is impossible, only that they wont do it.
You didn't even read this thread fully.
Quote from botg, 1st reply:
It is a technical impossibility
Boco, so right! It's been a long time since I read that, I totally forgot. I was wrong.

So yes, botg has claimed it's impossible (which it's clearly not), but he wont change his mind. No matter, we need alternatives or a fork, and I think a fork would be bad in this case. So above is my temp fix.

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: Is automatic uploading of changed files still wontfix?

#14 Post by botg » 2010-02-25 19:55

How do you solve the problem with a program doing a two-stage write with exclusive access?

Consider this sequence:
- Editor writes to file, closes file handle. But the work is not yet done, the file right now is inconsistent!
- Your script kicks in
- While file is still uploading, editor tries to re-open the file to finish the saving. Obviously it fails to obtain exclusive lock since your script is still uploading. Editor aborts.

Final result: Both the local file as well as the file on the server are inconsistent and cannot be opened anymore. In other words, all data has been lost.

jumpfroggy
503 Bad sequence of commands
Posts: 18
Joined: 2007-08-25 16:14

Re: Is automatic uploading of changed files still wontfix?

#15 Post by jumpfroggy » 2010-02-25 21:07

botg wrote:How do you solve the problem with a program doing a two-stage write with exclusive access?

Consider this sequence:
- Editor writes to file, closes file handle. But the work is not yet done, the file right now is inconsistent!
- Your script kicks in
- While file is still uploading, editor tries to re-open the file to finish the saving. Obviously it fails to obtain exclusive lock since your script is still uploading. Editor aborts.

Final result: Both the local file as well as the file on the server are inconsistent and cannot be opened anymore. In other words, all data has been lost.
Hi botg,

Very interesting situation! I'll try to dissect this below, please let me know if I get anything wrong. I'm just trying to understand this.

So, a two stage write with exclusive access. I totally agree that this could cause data loss. I believe the steps you listed expose bugs in the "editor". So rewritten in my own words, the steps would be:

1. Open a file exclusively for write.
2. Overwrite the file (ie. erase all existing content).
3. Possibly write some content, but not all of it.
4. Close the file.
5. Try to re-open the file exclusively for write.
6. If the file opened successfully, write the rest of the content (append?).
7. If the file could not be opened, fail w/o writing any data, resulting in data loss.

There seem to be multiple bugs in this situation. First off, the program is erasing existing content, then closing the file. This is a dangerous operation, since there's no guarantee that any subsequent file access will be possible, which also means there's no guarantee that the rest of the data will ever be written. I believe that's the crux of the matter here.

There's a very minot bug in step 7: if the editor can't open the file for writing again, fail w/o saving. Basically "If I can't write the data, then throw it away." There should be some error handling allowing the user to re-save a file, or some kind of graceful fallback. But this is a non issue, and even graceful fallback doesn't help due to steps 1 and 4.

If the existing contents were not erased in step 1, then any failure in step 7 would result in failure to save new content, but the old content would remain intact. Not ideal, but not as bad as the above situation.

And the huge issue: Step 4 & 5. "Close a file file I'm not done writing to, then re-open with exclusive write access." This is a huge mistake, and exposes a potential for data loss.

So I think I agree with you - this type of program would fail miserably. However, my opinion on the matter is different - this type of editor will fail with *any* program that reads the file at *any* time. If this type of editor is being used, there's absolutely no way for any other program on the system to know if it's safe to write a file, or if it's that magical time between steps 4 and 5.

A program like mine above would exacerbate the problem (due to more chance of triggering the bug), but essentially this type of editor loses data - there's no reason to use it. It could just as easily be triggered by something like TortoiseSVN which scans working-copy files for changes from the SVN respository, or windows indexing search.

I could be wrong about all this, and if so please correct me. It's very intersting to see a situation where data loss is possible just by reading the file, but it does seem that it's a buggy editor that's at fault, and not anything specific to an auto-sync program.

Post Reply