FileZilla 3 development diary

[botg]

The forums and especially this thread are not a place to request features. Use the feature request trackers for it.

[ariel]

I just wanted to suggest a commandline option for connecting to a site before i found your plans for that.

-c, --site=<str> Connect to specified Site Manager site

I'm really amazed to see this already being around the corner.

Just an idea: A function to create desktop icons/shortcuts that use the -c option to open filezilla already connected to a specific site. Maybe a 6th button in the site-manager called "Shortcut" could serve to create those desktop shortcuts.

Many thanks for this awesome program!!

Chris

In the new commandline feature for launching connections, it would be nice to be able to pass on a password over the command line, for those site-manager entries that require password but are not allowed to store it in the .xml file.

Thanks

[azuwis]

Or maybe I'll just stop offering FZ binaries for OS X. If Apple cannot provide me with a cross platform FLOSS tool to create icons, they don't want my program to run on OS X.

Have a look at http://icns.sourceforge.net/

[botg]

Have a look at http://icns.sourceforge.net/

Thanks, exactly what I was looking for.

[botg]

The most biggest major feature still missing in FileZilla 3 is support for IPv6.

Unfortunately, wxWidgets does not support IPv6 and will not have it any time soon. As such I have to search for an alternative.

I did a bit of searching to see what socket classes are out there and the outlook is bleak.

Some of the socket classes I found are truly overkill, like for example asio (part of boost) or this huge class library: http://www.alhem.net/Sockets/classdiagram.html
They are too complex and don't integrate that well into existing code. Note that I have to weave any new socket code into the event handling system of wxWidgets, not a pretty task.

Other socket classes basically have the same limitations as wxWidgets, e.g. no IPv6 support. Others don't run on all platforms FileZilla runs on. Last but not least some use an incompatible license or even the wrong programming language.

So the best option would possibly be to write my own socket class. I think I'll take the networking code in PuTTY as example, it is already cross-platform and supports IPv6. It is also very simple and easy to understand.

[botg]

IPv6 support will bump the version number of FileZilla to 3.1.0

I did some brainstorming and started to write the class declarations.

Writing socket classes is a very annoying task. Every serious developer working with networking applications ends up doing it sooner or later. It's now the fifth time I write a socket class. You can find one implementation in FZ 2.x and FZS.
The problem is that the socket classes in most frameworks attempt to be Jack of all trades, while actually being very limited. For example most if not all socket classes still support blocking IO. What's the point of blocking networking IO in a GUI framework? Exactly to frustrate the user with a user interface that is unresponsive during socket operations. On the other hand if you just have an asynchronous socket, even the most braindead VB.net pseudo-developer can make it blocking.
Most framework's socket classes are hard to extend with new functionality. For example in wxWidgets, most if not all of the interesting functionality is in private functions and there's no access to the raw socket either. (Trick: before including the header, do some #define magic to sneak in some more public: somewhere in the class declaration. Might cause linker errors.)
And on top of it, the socket code in wxWidgets is incredibly complex. Spread over dozens of files, with some shared files and some platform specific files. The platform specific files are somewhat similar but have some subtle differences.

I decided to create a class that is purely non-blocking. It's API will be somewhat similar to a subset of the wxSocket API, but much more simple. There will be one helper thread for every socket and select gets used to wait for socket events which in turn will be converted into wxWidgets events by the helper thread. The send event semantics will be mostly identical to the wxWidgets' one. However I'll change the read one to be just like the write one for increased performance.

I'd like to have a way to detach the socket and attach it to the file IO thread that does the reading and writing of files. That way I could do zero-copy transfers. For example on Linux see 'man 2 splice' or 'man 2 sendfile'. While this most likely won't make it into 3.1.0, I still have to keep this feature in mind or I would have to rework the new socket class once again.

Oh and before anybody asks a stupid question: If your question can be answered with "When it's done", don't ask.

[botg]

After a lot more brainstorming, prototyping and throwing away several ideas I have begun implementing the socket class.

So far about 400 lines of code which cannot yet do anything at all. Mostly empty stubs, thread initialization and the beginning of the name resolution using getaddrinfo.

Roadmap for the things ahead, in order:

• implement the select (and on Windows: WSAWaitForMultipleObjects) wrapper.
• Implement connect
• Up to this point, none of the existing code has been touched, but after this FileZilla will be non-functional until the class has been completely finished
• Rip out the old async host resolver code, not needed anymore
• Try connecting to a server using the new class. Note: Just connecting, not reading nor writing any data
• Implement read and write functionality
• Cleanup and some more error handling I couldn't do inbetween
• Spin a beta release
• Lots of debugging
• ...
• Profit!

[da chicken]

Just wanted to thank you, botg, for the excellent quality and continued development and support for FileZilla. It's truly an excellent program!

[botg]

One nasty issue of the different in error codes between proper systems and Windows.

Proper systems use the error codes defined by POSIX. Or at least that's what I think defines the error codes. Anyhow, virtually all Unix(-like) systems use these codes. I call them POSIX error codes for now (somebody correct me if I'm wrong please).
Windows on the other hand uses a crude mix of POSIX error codes and custom error codes.

Since my socket class will be proper, it has to map Windows error codes onto POSIX error codes. Often there's a 1-to-1 mapping, but not always. It's a simple, but very tedious and mechanical task. Frustration factor 10.

Example:

POSIX has EISCONN and EINPROGRESS (and many many more), Windows has WSAEISCONN and WSAEINPROGRESS.

POSIX has EAI_SYSTEM, Windows has no equivalent for it.

Windows has WSANOTINITIALISED, POSIX has no equivalent for it.

In some cases, there are semantical differences between the meaning of the error codes depending on platform. Needs call-specific conversion in some places. For example take connect():
On POSIX, EINPROGRESS means non-blocking operation is in progress and not yet complete. On Windows, WSAEINPROGRESS however means a blocking operation is in progress, the current call failed. The equivalent to EINPROGRESS would be WSAEWOULDBLOCK. Bit of a mess.

[botg]

I'm making good progress.

For the first time now I managed to connect to the server, get the directory listing and transfer a file. It's just hailing assertions the size of golfballs, active mode doesn't work and the FTP part of IPv6 doesn't work yet. And a giant memory leak from hell covers the fact that I currently don't even catch close notifications.

In other words: Walk along, nothing to see, it's all working as intended

[botg]

Active mode works again, SSL/TLS works again. More to come...

[botg]

A few versions ago I made the update check submit the operating system version number to the server, to prevent issues like the OS X 10.4 -> 10.5 update problem.

This also made it possible to collect some statistics about the userbase.

A random sampling of one million update checks shows the following distribution amongst Windows users:

Code: Select all

Windows NT 4:                  1 user
Windows 2000:                  1.47%
Windows XP:                   75.60%
Windows XP 64bit, Server 2003: 2.84% 
Windows Vista, Server 2008:   20.00%
Windows 7:                     3 users

Looks like somebody working at Microsoft likes FileZilla, why else would there be Windows 7

It clearly shows that Windows 2000 users are a minority. This is of relevance, since Windows 2000 does not have an API function I need for my socket class, as somebody kindly brought to my attention.
Personally I have no interest in wasting lots of time supporting a long deprecated operating system, so unless somebody else takes care of it, I'll drop Win2000 support.

[GoD]

Can you give a little bit more infos?
Can you differentiate between Vista x32 and Vista x64? The same for Server 2008?
Would be interesting to see the popularity of x64 windows systems in a graph over
the next months

[botg]

Only thing I can see is the NT version number, see http://en.wikipedia.org/wiki/Windows_NT#Releases

[boco]

What I really want to know is how the Windows NT 4 user got there. Self compiled? In that case an update check wouldn't make any sense, don't 'ya think?