FileZilla 3 development diary
Moderator: Project members
Re: FileZilla 3 development diary
Some parts of FileZilla are once again starting to become unmaintainable. Every new feature adds complexity and the code of some of the more important classes is slowly turning into overcooked spaghetti again, loosing all structure.
My gripe this time is tracking user interface setting changes. The way it is done works like this:
Some component changes a setting and explicitly tells all other involved components to update their state. Needless to say this quickly gets complex if more than two components depend on the same setting.
What I want to see instead is some sort of observer or listener pattern. That is each component interested in a particular setting registers at the options class, which in turn will simply notify all handlers when a setting changes.
Something similar to this is already implemented in the context state class, i.e. the class that tracks context variables, such as local and remote directories, connected server, used engine and so on. More on contexts, FileZilla uses a context for each tab. To be specific, it associates a control container with each context. The GUI stuff on the control container, the non-GUI stuff in the context.
Back to the option observers, it shouldn't be hard to implement, just tedious and time consuming. But in the end worth the effort I hope. Especially that 3360 line monster that is Mainfrm.cpp would greatly benefit from it if lots of code gets moved to a more appropriate place.
My gripe this time is tracking user interface setting changes. The way it is done works like this:
Some component changes a setting and explicitly tells all other involved components to update their state. Needless to say this quickly gets complex if more than two components depend on the same setting.
What I want to see instead is some sort of observer or listener pattern. That is each component interested in a particular setting registers at the options class, which in turn will simply notify all handlers when a setting changes.
Something similar to this is already implemented in the context state class, i.e. the class that tracks context variables, such as local and remote directories, connected server, used engine and so on. More on contexts, FileZilla uses a context for each tab. To be specific, it associates a control container with each context. The GUI stuff on the control container, the non-GUI stuff in the context.
Back to the option observers, it shouldn't be hard to implement, just tedious and time consuming. But in the end worth the effort I hope. Especially that 3360 line monster that is Mainfrm.cpp would greatly benefit from it if lots of code gets moved to a more appropriate place.
-
- 226 Transfer OK
- Posts: 392
- Joined: 2008-12-30 10:30
- First name: John
- Last name: Ratliff
- Location: In a small white padded room.
Re: FileZilla 3 development diary
Have you looked at wxUpdateUIEvent?
http://jdrrant.blogspot.com/ - CODEpendent Blog
Re: FileZilla 3 development diary
Yes, and it's horrible. Those update UI events get triggered constantly, all the time. And if they get triggered, all get triggered, no way to filter. Just moving the mouse over the main window would consume lots of CPU.Have you looked at wxUpdateUIEvent?
That's why I eventually switched away from them to the current system, updating everything manually. UI update events have been disabled in FZ, see src/interface/FileZilla.cpp:279
Re: FileZilla 3 development diary
This weekend I've done some cleanup work, but before I can explain what I've done let me start by a small digression into wxWidgets:
One of the many classes offered by wxWidgets is wxFrame, which is derived from wxTopLevelWindow which is derived from wxWindow. A frame is a special type of top level window, with the other important type of top level window being the dialogs.
Back on topic, I've worked on CMainFrame, a class derived from wxFrame that is responsible for the main window of FileZilla. It's a huge beast of over 3300 lines of code. The problem is that CMainFrame is a God object, it does too much. Not only does it create (and on shutdown destroy) all other child windows, it also handles menu bar and tool bar events and updates the state of some other GUI elements.
To alleviate this problem I've implemented the option change observer I mentioned recently. I started by moving large parts of the toolbar handling into a new class.
In the next step I tried to move the menu bar related code into its own class but ran into a problem. FileZilla loads the layout of its GUI components from XML resource files (.xrc). Since toolbars and menu bars do not support 2-stage creation, I need to use subclassing. For toolbars this works fine, but unfortunately subclassing for wxMenuBar does not work. Didn't take me long to create a patch for this issue.
The good news, the release candidate of wxWidgets 2.8.11 has been released. I very much hope that this patch makes it into the final 2.8.11 release of wx.
The bad news, the package maintainers of the various Linux distributions and other operating systems will probably kill me for once again requiring such bleeding edge dependencies
One of the many classes offered by wxWidgets is wxFrame, which is derived from wxTopLevelWindow which is derived from wxWindow. A frame is a special type of top level window, with the other important type of top level window being the dialogs.
Back on topic, I've worked on CMainFrame, a class derived from wxFrame that is responsible for the main window of FileZilla. It's a huge beast of over 3300 lines of code. The problem is that CMainFrame is a God object, it does too much. Not only does it create (and on shutdown destroy) all other child windows, it also handles menu bar and tool bar events and updates the state of some other GUI elements.
To alleviate this problem I've implemented the option change observer I mentioned recently. I started by moving large parts of the toolbar handling into a new class.
In the next step I tried to move the menu bar related code into its own class but ran into a problem. FileZilla loads the layout of its GUI components from XML resource files (.xrc). Since toolbars and menu bars do not support 2-stage creation, I need to use subclassing. For toolbars this works fine, but unfortunately subclassing for wxMenuBar does not work. Didn't take me long to create a patch for this issue.
The good news, the release candidate of wxWidgets 2.8.11 has been released. I very much hope that this patch makes it into the final 2.8.11 release of wx.
The bad news, the package maintainers of the various Linux distributions and other operating systems will probably kill me for once again requiring such bleeding edge dependencies
- judahrad
- 500 Command not understood
- Posts: 1
- Joined: 2010-07-08 02:34
- First name: Judah
- Last name: Rad
Re: FileZilla 3 development diary
Using Filezilla as mt FTP makes my work faster.. i Really Love it..
-
- 421 Kicked by Administrator
- Posts: 1
- Joined: 2010-08-02 12:08
- First name: Amelia
- Last name: Johnson
Re: FileZilla 3 development diary
I agree it's a fabulous program, so easy to use, i can't see what all the fuss is about on this forum, but i guess it can get complicated if things don't work like that should.judahrad wrote:Using Filezilla as mt FTP makes my work faster.. i Really Love it..
Last edited by neubree739 on 2011-02-21 04:31, edited 1 time in total.
Re: FileZilla 3 development diary
Some news:
In an effort to increase security caused by malware sniffing FTP passwords straight from the network adapter, I am making FTP over TLS the default protocol. This will be done in two steps:
In an effort to increase security caused by malware sniffing FTP passwords straight from the network adapter, I am making FTP over TLS the default protocol. This will be done in two steps:
- Always attempt AUTH TLS when connecting. If it fails, print an option into the log but keep on connecting.
- In the future, abort connection if AUTH TLS fails, lest we send the password in the clear.
-
- 500 Command not understood
- Posts: 1
- Joined: 2011-02-09 05:59
- First name: Vb
- Last name: Reader
Re: FileZilla 3 development diary
I love the FileZilla bookmarks feature. I am using filezilla for 4 of my websites to publish .Net website. Is there any way i can upload the files to all four ftp sites at one upload(like a batch). This will be a great addition for multiple website publishers.
Sorry if this feature is already available,if there please direct me.
Thank you for giving us a great application.
Sorry if this feature is already available,if there please direct me.
Thank you for giving us a great application.
Re: FileZilla 3 development diary
The queue is independent from how many sites you have opened. No matter how you queue files for upload, they are processed sequentially in a FIFO manner (unless you change priorities).
If you mean automatic batch-like transfers, no, FileZilla cannot do that.
If you mean automatic batch-like transfers, no, FileZilla cannot do that.
No support requests over PM! You will NOT get any reply!!!
FTP connection problems? Please read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
FileZilla Pro support: https://customerforum.fileZilla-project.org
FTP connection problems? Please read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
FileZilla Pro support: https://customerforum.fileZilla-project.org
Re: FileZilla 3 development diary
I have read all the pages. From page 1 to 63. You get bored of writing on 2010?
It was fun and enlightening reading all this though!
It was fun and enlightening reading all this though!
Re: FileZilla 3 development diary
Not bored, just busy
Currently I'm working on using SQLite to store the transfer queue on disk. Rationale being the low performance and excessive memory consumption of the used XML parser. Queues of a few million files were unsaveable as it would have required several GiB of RAM. With SQLite, saving the queue is fast, feels almost instant, and requires a negligible amount of memory. Expect the first bits of code to hit the repository soon.
In addition, I'm working on a string coalescer. The string class in wxWidgets uses Copy-on-write. If a string is copied, only a pointer is copied and a reference counter increased. Only when a string is modified, a real copy is made (if reference count > 1). This helps in reducing memory footprint.
Example:
Both foo and bar now point to the same memory location. Now imagine foo containing a large string, such as a long path in a deeply nested directory tree. The savings can be vast.
However, it does not work for strings with unrelated origin:
In this case, foo and baz point to different memory locations.
To merge them, the following trick can be used:
The general idea is to have a container with already known strings and to coalesce them in strategic places.
The container has the following requirements:
Operations:
Sidenode: One could argue that, due to having a fixed size limit, the entire container is O(1). If the size limit is x bytes, no string with length k > x can be in the container and thus all operations are bound by O(k) < O(x) and due to x being constant, O(1). However, for analyzing this algorithm we assume that x is typically much much larger than k.
Currently I'm working on using SQLite to store the transfer queue on disk. Rationale being the low performance and excessive memory consumption of the used XML parser. Queues of a few million files were unsaveable as it would have required several GiB of RAM. With SQLite, saving the queue is fast, feels almost instant, and requires a negligible amount of memory. Expect the first bits of code to hit the repository soon.
In addition, I'm working on a string coalescer. The string class in wxWidgets uses Copy-on-write. If a string is copied, only a pointer is copied and a reference counter increased. Only when a string is modified, a real copy is made (if reference count > 1). This helps in reducing memory footprint.
Example:
Code: Select all
wxString foo = _T("Bar");
wxString baz = foo;
However, it does not work for strings with unrelated origin:
Code: Select all
wxString foo = _T("Bar");
wxString baz = _T("Bar");
To merge them, the following trick can be used:
Code: Select all
if (foo == baz) { baz = foo; }
The container has the following requirements:
- Fast, ideally O(k) with k being the length of the individual string. For comparison, a simple binary tree is O(log(n) * k) with n being number of items. This is due to trees being logarithmic only in the number of comparisons for lookup/insertion/deletions, however string comparison isn't O(1), it's O(k) in string length.
- Memory consumption limit, let's say no more than 100 MB total. Thus the number of items that can be in it is limited and also depends of the length of the individual items.
Operations:
- Insertion in O(k):
Whenever a new item is inserted, the pointer to the node in the trie is appended to the end of the LRU and the list iterator remembered in the trie node. If the container becomes too large, items are pruned from the start of the LRU list as well as their corresponding trie entries. Inserting into trie is O(k), appending to the list O(1). Hence O(k). - Lookup in O(k):
Whenever an item is looked up and in the container, the LRU is updated. The item in the list is simply moved to the back. Trie lookup can be done in O(k) and moving an item from a known position in a list to the end is O(1).
Sidenode: One could argue that, due to having a fixed size limit, the entire container is O(1). If the size limit is x bytes, no string with length k > x can be in the container and thus all operations are bound by O(k) < O(x) and due to x being constant, O(1). However, for analyzing this algorithm we assume that x is typically much much larger than k.
Re: FileZilla 3 development diary
The SQLite backend for storing the queue has landed in the repository. Still lots of rough edges and corners, but so far it's working amazingly great. Saving and loading of huge queues is now very fast and no longer consumes a few dozen GiB of RAM.
And I still have ideas how to further reduce save times. The re-binding of values can be avoided if they are identical. In particular, directories commonly contain multiple files. Thus, for all files in the same directory, the local and remote directories are identical and don't need to be rebinded. Checking for that trivially cheap, simple pointer comparison due to having (soon) the coalescer I wrote about in my previous post.
Last but not least, I think it's not necessary to save both the local and remote file name if they are identical. To save diskspace and memory, one can be left NULL easily.
And I still have ideas how to further reduce save times. The re-binding of values can be avoided if they are identical. In particular, directories commonly contain multiple files. Thus, for all files in the same directory, the local and remote directories are identical and don't need to be rebinded. Checking for that trivially cheap, simple pointer comparison due to having (soon) the coalescer I wrote about in my previous post.
Last but not least, I think it's not necessary to save both the local and remote file name if they are identical. To save diskspace and memory, one can be left NULL easily.
Re: FileZilla 3 development diary
A first version of the string coalescer has been committed to the repository. For now it is using a hashmap as tree.
Since FileZilla is Unicode enabled, it uses wide-character strings: UTF-16 on Windows, Unicode codepoints on other platforms. Using a simple trie, the node fan-out would be quite big. So far I could not find a suitable trie implementation for this task, so instead I opted for a simple hash map for the tree, in the form of C++ TR1's unordered_map. Performance is reasonably well and due to the efficient hash function, operations still take amortized O(k) with k being string length. Worst-case though will be O(n) with n being the number of items, if one were to specifically craft input strings to cause hash function collisions.
Some measured statistics of the string coalescer running under Windows on an Core i7 Q720:
Since FileZilla is Unicode enabled, it uses wide-character strings: UTF-16 on Windows, Unicode codepoints on other platforms. Using a simple trie, the node fan-out would be quite big. So far I could not find a suitable trie implementation for this task, so instead I opted for a simple hash map for the tree, in the form of C++ TR1's unordered_map. Performance is reasonably well and due to the efficient hash function, operations still take amortized O(k) with k being string length. Worst-case though will be O(n) with n being the number of items, if one were to specifically craft input strings to cause hash function collisions.
Some measured statistics of the string coalescer running under Windows on an Core i7 Q720:
The entries have been generated by queuing up a few hundred thousand files. Without the coalescer, FileZilla would have been unable to load such queues on Windows due to the per-process 2GiB memory limit for 32bit processes. With the coalescer, overall memory usage is at 267 MiB. That number will go further down though as I have not yet exhausted all optimization potential.Looking up all strings a hundred times took 1139ms.
Entries: 23154
Assumed memory consumption: 3202364
Average string length: 48.154
Median string length: 51
Re: FileZilla 3 development diary
Would love to test it, but all Nightly builds currently fail.
configure: error: sqlite3.h not found which is part of SQLite3.
No support requests over PM! You will NOT get any reply!!!
FTP connection problems? Please read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
FileZilla Pro support: https://customerforum.fileZilla-project.org
FTP connection problems? Please read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
FileZilla Pro support: https://customerforum.fileZilla-project.org
Re: FileZilla 3 development diary
Not testworthy yet, still too many rough edges
Fully aware that it doesn't compile yet, didn't install the required dependencies yet on the build slaves.
Fully aware that it doesn't compile yet, didn't install the required dependencies yet on the build slaves.