FileZilla 3 development diary

[botg]

Some parts of FileZilla are once again starting to become unmaintainable. Every new feature adds complexity and the code of some of the more important classes is slowly turning into overcooked spaghetti again, loosing all structure.

My gripe this time is tracking user interface setting changes. The way it is done works like this:
Some component changes a setting and explicitly tells all other involved components to update their state. Needless to say this quickly gets complex if more than two components depend on the same setting.

What I want to see instead is some sort of observer or listener pattern. That is each component interested in a particular setting registers at the options class, which in turn will simply notify all handlers when a setting changes.

Something similar to this is already implemented in the context state class, i.e. the class that tracks context variables, such as local and remote directories, connected server, used engine and so on. More on contexts, FileZilla uses a context for each tab. To be specific, it associates a control container with each context. The GUI stuff on the control container, the non-GUI stuff in the context.

Back to the option observers, it shouldn't be hard to implement, just tedious and time consuming. But in the end worth the effort I hope. Especially that 3360 line monster that is Mainfrm.cpp would greatly benefit from it if lots of code gets moved to a more appropriate place.

[jdratlif]

Have you looked at wxUpdateUIEvent?

[botg]

Have you looked at wxUpdateUIEvent?

Yes, and it's horrible. Those update UI events get triggered constantly, all the time. And if they get triggered, all get triggered, no way to filter. Just moving the mouse over the main window would consume lots of CPU.

That's why I eventually switched away from them to the current system, updating everything manually. UI update events have been disabled in FZ, see src/interface/FileZilla.cpp:279

[botg]

This weekend I've done some cleanup work, but before I can explain what I've done let me start by a small digression into wxWidgets:
One of the many classes offered by wxWidgets is wxFrame, which is derived from wxTopLevelWindow which is derived from wxWindow. A frame is a special type of top level window, with the other important type of top level window being the dialogs.

Back on topic, I've worked on CMainFrame, a class derived from wxFrame that is responsible for the main window of FileZilla. It's a huge beast of over 3300 lines of code. The problem is that CMainFrame is a God object, it does too much. Not only does it create (and on shutdown destroy) all other child windows, it also handles menu bar and tool bar events and updates the state of some other GUI elements.

To alleviate this problem I've implemented the option change observer I mentioned recently. I started by moving large parts of the toolbar handling into a new class.

In the next step I tried to move the menu bar related code into its own class but ran into a problem. FileZilla loads the layout of its GUI components from XML resource files (.xrc). Since toolbars and menu bars do not support 2-stage creation, I need to use subclassing. For toolbars this works fine, but unfortunately subclassing for wxMenuBar does not work. Didn't take me long to create a patch for this issue.

The good news, the release candidate of wxWidgets 2.8.11 has been released. I very much hope that this patch makes it into the final 2.8.11 release of wx.
The bad news, the package maintainers of the various Linux distributions and other operating systems will probably kill me for once again requiring such bleeding edge dependencies

[judahrad]

Using Filezilla as mt FTP makes my work faster.. i Really Love it..

[neubree739]

Using Filezilla as mt FTP makes my work faster.. i Really Love it..

I agree it's a fabulous program, so easy to use, i can't see what all the fuss is about on this forum, but i guess it can get complicated if things don't work like that should.

[botg]

Some news:

In an effort to increase security caused by malware sniffing FTP passwords straight from the network adapter, I am making FTP over TLS the default protocol. This will be done in two steps:

1. Always attempt AUTH TLS when connecting. If it fails, print an option into the log but keep on connecting.
2. In the future, abort connection if AUTH TLS fails, lest we send the password in the clear.

Of course, people will have the choice to explicitly make use of insecure plain FTP through the Site Manager. During phase 1, users will only need this if the server is behind those horribly broken routers and/or firewalls that sabotage FTP deliberately.

[vb_reader]

I love the FileZilla bookmarks feature. I am using filezilla for 4 of my websites to publish .Net website. Is there any way i can upload the files to all four ftp sites at one upload(like a batch). This will be a great addition for multiple website publishers.

Sorry if this feature is already available,if there please direct me.

Thank you for giving us a great application.

[boco]

The queue is independent from how many sites you have opened. No matter how you queue files for upload, they are processed sequentially in a FIFO manner (unless you change priorities).

If you mean automatic batch-like transfers, no, FileZilla cannot do that.

[luislobo]

I have read all the pages. From page 1 to 63. You get bored of writing on 2010?

It was fun and enlightening reading all this though!

[botg]

Not bored, just busy

Currently I'm working on using SQLite to store the transfer queue on disk. Rationale being the low performance and excessive memory consumption of the used XML parser. Queues of a few million files were unsaveable as it would have required several GiB of RAM. With SQLite, saving the queue is fast, feels almost instant, and requires a negligible amount of memory. Expect the first bits of code to hit the repository soon.


In addition, I'm working on a string coalescer. The string class in wxWidgets uses Copy-on-write. If a string is copied, only a pointer is copied and a reference counter increased. Only when a string is modified, a real copy is made (if reference count > 1). This helps in reducing memory footprint.

Example:

Code: Select all

wxString foo = _T("Bar");
wxString baz = foo;

Both foo and bar now point to the same memory location. Now imagine foo containing a large string, such as a long path in a deeply nested directory tree. The savings can be vast.

However, it does not work for strings with unrelated origin:

Code: Select all

wxString foo = _T("Bar");
wxString baz = _T("Bar");

In this case, foo and baz point to different memory locations.

To merge them, the following trick can be used:

Code: Select all

if (foo == baz) { baz = foo; }

The general idea is to have a container with already known strings and to coalesce them in strategic places.

The container has the following requirements:

• Fast, ideally O(k) with k being the length of the individual string. For comparison, a simple binary tree is O(log(n) * k) with n being number of items. This is due to trees being logarithmic only in the number of comparisons for lookup/insertion/deletions, however string comparison isn't O(1), it's O(k) in string length.
• Memory consumption limit, let's say no more than 100 MB total. Thus the number of items that can be in it is limited and also depends of the length of the individual items.

A data structure that would work for this is trie (aka prefix tree) combined with a linked list as LRU list. The leaf nodes in the trie contain the string and an iterator into the LRU list. The list entries are simply pointers to the trie leaf nodes.

Operations:

• Insertion in O(k):
  Whenever a new item is inserted, the pointer to the node in the trie is appended to the end of the LRU and the list iterator remembered in the trie node. If the container becomes too large, items are pruned from the start of the LRU list as well as their corresponding trie entries. Inserting into trie is O(k), appending to the list O(1). Hence O(k).
• Lookup in O(k):
  Whenever an item is looked up and in the container, the LRU is updated. The item in the list is simply moved to the back. Trie lookup can be done in O(k) and moving an item from a known position in a list to the end is O(1).

It should be noted that string lengths are measured when creating strings, thus with wxString, getting string length is O(1). The memory overhead of trie and list entries is a constant known already at compile time and thus it is efficient to dynamically calculate the total size of the container and do the LRU pruning.



Sidenode: One could argue that, due to having a fixed size limit, the entire container is O(1). If the size limit is x bytes, no string with length k > x can be in the container and thus all operations are bound by O(k) < O(x) and due to x being constant, O(1). However, for analyzing this algorithm we assume that x is typically much much larger than k.

[botg]

The SQLite backend for storing the queue has landed in the repository. Still lots of rough edges and corners, but so far it's working amazingly great. Saving and loading of huge queues is now very fast and no longer consumes a few dozen GiB of RAM.

And I still have ideas how to further reduce save times. The re-binding of values can be avoided if they are identical. In particular, directories commonly contain multiple files. Thus, for all files in the same directory, the local and remote directories are identical and don't need to be rebinded. Checking for that trivially cheap, simple pointer comparison due to having (soon) the coalescer I wrote about in my previous post.

Last but not least, I think it's not necessary to save both the local and remote file name if they are identical. To save diskspace and memory, one can be left NULL easily.

[botg]

A first version of the string coalescer has been committed to the repository. For now it is using a hashmap as tree.

Since FileZilla is Unicode enabled, it uses wide-character strings: UTF-16 on Windows, Unicode codepoints on other platforms. Using a simple trie, the node fan-out would be quite big. So far I could not find a suitable trie implementation for this task, so instead I opted for a simple hash map for the tree, in the form of C++ TR1's unordered_map. Performance is reasonably well and due to the efficient hash function, operations still take amortized O(k) with k being string length. Worst-case though will be O(n) with n being the number of items, if one were to specifically craft input strings to cause hash function collisions.

Some measured statistics of the string coalescer running under Windows on an Core i7 Q720:

Looking up all strings a hundred times took 1139ms.

Entries: 23154
Assumed memory consumption: 3202364
Average string length: 48.154
Median string length: 51

The entries have been generated by queuing up a few hundred thousand files. Without the coalescer, FileZilla would have been unable to load such queues on Windows due to the per-process 2GiB memory limit for 32bit processes. With the coalescer, overall memory usage is at 267 MiB. That number will go further down though as I have not yet exhausted all optimization potential.

[boco]

Would love to test it, but all Nightly builds currently fail.

configure: error: sqlite3.h not found which is part of SQLite3.

[botg]

Not testworthy yet, still too many rough edges

Fully aware that it doesn't compile yet, didn't install the required dependencies yet on the build slaves.