FileZilla 3 development diary

Moderator: Project members

Message
Author
da chicken
226 Transfer OK
Posts: 619
Joined: 2005-11-02 06:41

Re: FileZilla 3 development diary

#856 Post by da chicken » 2008-12-29 08:46

botg wrote:Ever tried OS X? It has built-in virtualization and emulation. Run PPC binaries on Intel Macs or run binaries for OS 9 on OS X (earlier PPC versions at least, not supported anymore afaik).

Microsoft could easily do the same with the next Windows version. A fully backwards compatible VM layer for old binaries, and the rest of the system KDE ;)
Windows already does this, and has for years. Under 2000 and later, 16-bit Windows apps run in WoW (Windows on Windows), and DOS applications run in the NTVDM (NT Virtual DOS Machine). On 64-bit Windows, 32-bit applications run in WOW64. It's generally not considered "virtualization" as much as API emulation or wrapping.

The underlying problem isn't even with the window manager. It's with the file system and the way the OS kernel resolves file deletions for shared files. Your problem is with the OS. What have KDE and GNOME got to do with the OS? Even if you ran GNOME or KDE as the window manager on Windows, you'd still have the NT kernel and NTFS limitations beneath that. Yes, I'd be happy if Windows 7 came out running with support for an ext3 inspired file system with NTFS permissions, auditing, and such added on to it because, yes, the NTFS and FAT limitations are horrible. I'm convinced that FAT is probably one of the reasons that the behavior is so bad. Yes, they ought to rewrite the whole thing and then remap the functionality of these old file sharing flags to use better functions rather like WINE (or wxWidgets, or any other toolkit) does, but they haven't done that. Maybe they have in .NET. I don't know.

mno
500 Syntax error
Posts: 12
Joined: 2008-02-15 17:39

Re: FileZilla 3 development diary

#857 Post by mno » 2008-12-29 08:51

da chicken wrote:Yes, they ought to rewrite the whole thing and then remap the functionality of these old file sharing flags to use better functions rather like WINE (or wxWidgets, or any other toolkit) does, but they haven't done that. Maybe they have in .NET. I don't know.
They have some nice wrappers in .NET for reading, writing, deleting files. However, I don't know how they handle shared deletions there. It may be that once you delete, to you, it seems like the file is gone, but underlying, the framework still has to work out all the issues with the kernel and filesystem. So I doubt they "solved" it in .NET. For that matter, I doubt anything is "solvable" in .NET as it still relies on the kernel and filesystem...

User avatar
botg
Site Admin
Posts: 35509
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#858 Post by botg » 2009-01-17 21:22

While FileZilla 3.2.0 has been a great release (i.e. no regressions), it's still far from feature complete.

Not a single day goes by without at least a dozen users requesting new features. These features range from highly useful (Please add support for example:// protocol) to utterly useless (Our server admins are stupid, please add option to disallow users to access certain directories with FileZilla), from easy (Add a confirmation dialog for some actions) over hard (example:// protocol) to impossible (I want to use active mode behind NAT without having to configure my router).

Worst part of it, it's all duplicates. I'm rarely getting a feature request that hasn't already been requested long time ago. Wish I could clone myself...

Now for the next version of FileZilla I'm adding a feature which has been requested many times over and over again: Synchronized browsing, sometimes also called lockstep navigation or tandem mode.
It's quite a useful feature, though I probably won't ever use it myself, my directory hierarchies are rather flat and under version control.

At first glance implementing this appears to be an easy task. But as so often, it's much more complex than it looks at first. (Simple analogue: If you are read this, you're using a computer, an easy task. But do you truly understand your computer in detail? Could you build and program a machine from scratch?).
For example I need a function that, given a local and a remote directory, returns the most common parents of both paths with as many identical suffixes removed as possible.
In particular I'm missing a good class to encapsulate the local path syntax and operations on it (HasParent(), GetParent() and so on). Sure there's already wxWidgets' own wxFileName and wxDir classes. Why am I not using them? Simple: Their APIs are unintuitive and first and foremost, they are slow. For example wxDir has methods to list the contents of a local directory. I went through great lengths to avoid using wxDir (see src/engine/local_filesys.cpp) simply to speed things up considerably by using a bare-metal approach in performance critical code paths. Doing some easy operations with wxFileName and wxDir involved multiple calls to the system's filesystem API which resulted in horrible performance if accessing directories mounted from remote shares. Now it only uses a single call.

How to fix this? As I have probably mentioned before, there already is CServerPath in FileZilla. Basically this class encapsulates the server's path syntax. It essentially has support for different server types, amongst those UNIX-like (/foo/bar/baz) and DOS-like (C:\foo\bar\baz).
Basically all I have to is to copy CServerPath, name it CLocalPath and clamp it to a specific type (i.e. DOS on Windows, UNIX everywhere else).

So the next task ahead is to write CLocalPath, convert as much existing code in FZ to use it and only then I can begin to actually implement the synchronized browsing.

eyebex
226 Transfer OK
Posts: 171
Joined: 2004-04-02 15:24

Re: FileZilla 3 development diary

#859 Post by eyebex » 2009-01-26 09:35

As a follow-up to the OS X icon discussion, here's an online conversion tool that might be of use: http://www.eisbox.net/iconvert/

User avatar
botg
Site Admin
Posts: 35509
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#860 Post by botg » 2009-01-30 15:24

Synchronized browsing has been implemented. Expect 3.2.1-rc1 soon.

mno
500 Syntax error
Posts: 12
Joined: 2008-02-15 17:39

Re: FileZilla 3 development diary

#861 Post by mno » 2009-01-30 15:29

Yay! :D

DavidONE
500 Syntax error
Posts: 15
Joined: 2008-06-10 17:04

Re: FileZilla 3 development diary

#862 Post by DavidONE » 2009-02-01 16:13

> Synchronized browsing has been implemented.

Excellent. It works well, but could it be made less intrusive? Can we have an option to turn off the nag every time synchronisation is lost? Thanks.

User avatar
botg
Site Admin
Posts: 35509
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#863 Post by botg » 2009-02-25 21:03

Perhaps the most commonly requested feature nowadays is automation support. While automation is a nice idea, I think it's horribly wrong to automate GUI programs when a dedicated commandline tool can perform the task way better.

Perhaps the best solution would be to create a new commandline version of FileZilla. For the start I think some small tools should suffice, something along the lines of fzget, fzput, fzls.

Before I could even start working on those tools I'd have to do some major changes to the FileZilla internals. In particular what's currently known as the engine (everything under src/engine) needs to be transformed into a stand-alone (more or less, the wx dependency isn't going away any time soon) libfilezilla.

The easy task is splitting off the code. Hardest will be updating the whole build system. Also, I'd then be forced to maintainthings like ABI compatibility for future releases. Another nightmare would be the administrative work surrounding such a feature. I'd would have to rework the release management scripts and a plethora of scripts running behind the scenes on filezilla-project.org

Last but not least, the package maintainers of the various Linux distributions will have lots of fun with such a change ;)

Anyhow, don't expect any miracles, implementing these huge changes would take a lot of time. Most likely several "normal" versions will be released in the meantime. On the bright side, I'll never run out of work, no time for boredom ;)

User avatar
boco
Contributor
Posts: 26913
Joined: 2006-05-01 03:28
Location: Germany

Re: FileZilla 3 development diary

#864 Post by boco » 2009-02-27 08:23

Giving you two thumbs up for the feature (I don't have more thumbs :mrgreen:)!
No support requests over PM! You will NOT get any reply!!!
FTP connection problems? Please read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
FileZilla Pro support: https://customerforum.fileZilla-project.org

User avatar
botg
Site Admin
Posts: 35509
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#865 Post by botg » 2009-03-28 18:43

After much deliberation I've finally decided to enable use of MLSD for directory listings as defined in RFC 3659.

Doing so has several benefits:
  • Well-defined syntax of returned listings. The listings returned by LIST on the other hand are not standardized in any way. Even with listings that resemble Unix' "ls -l" output there is a lot of variation and sometimes unresolvable ambiguities. For example consider the following, abbreviated date format: 04/05/06. There are six possible ways to interpret it and sadly I have seen them all used in practice.
    A lot of code went into FileZilla so that it can parse the most common listing formats as well as several rather exotic ones (MVS for example, its path and listing syntax is sadistic to put it lightly).
  • Timestamps in MLSD listings are required to have seconds resolution. Common "ls -l" rarely has seconds and often not even time (on older files).
  • Being able to detect leading and trailing spaces in filenames which isn't possible using LIST due to unspecified padding.
But where there's light there's shadow. Let's ignore the fact that many servers do not (yet) support MLSD, on those FZ will just fall back to the current behavior of using LIST. The problem stems from broken servers that hide files from the user. A proper server always shows all files and it's up to the used program to filter out files the user does not want to see. The important part is that the user has to remain in control.
One incredibly wonky hack employed by FileZilla is the use of the highly unstandardized, RFC violating "LIST -a" command. Problem again with that is that it's not supported by all servers which could sometimes result in strange behavior.
MLSD has no -a argument and there will never be one. If some servers hides files with MLSD there is only one solution: Server needs to be fixed. I will not work around this problem, period.

Another slight inconvenience with MLSD is that it doesn't define facts for file owner and group. Some servers use UNIX.user, others UNIX.owner and still others UNIX.uid as factname for the file owner. FZ just has to try to recognize them all.

da chicken
226 Transfer OK
Posts: 619
Joined: 2005-11-02 06:41

Re: FileZilla 3 development diary

#866 Post by da chicken » 2009-04-10 08:56

I'd like it if the batch programs would mirror the options of other existing programs. That is, use the same syntax as wget, curl, or lftp if possible. I use 7-zip from the command line quite a bit on Windows for archiving logs, and it pisses me off constantly that the behavior and flags are different than those for tar.

You also may want to look at the way that WinSCP does batch command processing.

User avatar
botg
Site Admin
Posts: 35509
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#867 Post by botg » 2009-04-18 09:05

I had some time to look into ways to make create a commandline version of FileZilla. Unfortunately right now it is next to impossible.

The primary reason is that wxWidgets 2.8 lacks an event loop for console apps. I could work around this with some clever thread usage, but then the resulting program would as far as I can tell still depend on a running X server even though it only displays output on the console.

What I need is wxWidgets 3 but it's still far away, soon first unstable development release (2.9.0) will be released. wx 3 does have a console event loop. The most challenging task will be porting FileZilla over to wx 2.9/3, the changes are significant.

valterchaves
500 Command not understood
Posts: 3
Joined: 2009-04-18 18:40
First name: valter
Last name: chaves

Re: FileZilla 3 development diary

#868 Post by valterchaves » 2009-04-18 19:12

botg wrote:I had some time to look into ways to make create a commandline version of FileZilla. Unfortunately right now it is next to impossible.

The primary reason is that wxWidgets 2.8 lacks an event loop for console apps. I could work around this with some clever thread usage, but then the resulting program would as far as I can tell still depend on a running X server even though it only displays output on the console.

What I need is wxWidgets 3 but it's still far away, soon first unstable development release (2.9.0) will be released. wx 3 does have a console event loop. The most challenging task will be porting FileZilla over to wx 2.9/3, the changes are significant.

If the commandline version seems to be ab bit far from now, how about getting back to the idea of implementig automation support through the gui interface of filezilla?
Last edited by boco on 2009-04-18 21:59, edited 1 time in total.
Reason: Removed possible spam signature

User avatar
botg
Site Admin
Posts: 35509
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#869 Post by botg » 2009-04-18 22:43

valterchaves wrote:If the commandline version seems to be ab bit far from now, how about getting back to the idea of implementig automation support through the gui interface of filezilla?
No, I'd rather wait a few years until wx 3 stabilizes.

User avatar
botg
Site Admin
Posts: 35509
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#870 Post by botg » 2009-05-09 10:18

The next big feature I'm implementing is remote file search.

The search dialog will somewhat look like this mockup I've created in Gimp:
search_mockup.png
search_mockup.png (10.44 KiB) Viewed 8844 times
For the conditions I can reuse the existing code in the filter edit dialog.

So before I can start, I'll have to refactor the existing filter code to be more modular, in particular creating a new dedicated class for the control that displays the conditions.

Post Reply