FileZilla 3 development diary

Moderator: Project members

Message
Author
User avatar
botg
Site Admin
Posts: 31574
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: FileZilla 3 development diary

#991 Post by botg » 2015-06-22 07:27

Preview of a new feature

There are a number of recursive operations in FileZilla, for example when adding a remote directory tree to the queue. Currently, during these operations FileZilla navigates the remote directory hierarchy, entering and displaying each directory it visits. It is something that has worked well for over 14 years now.

There are however a number of problems with this approach:
  • While FileZilla does its work, the user cannot do anything else in the current tab
  • Displaying each directory as it is visited is slow, it consumes a lot of valuable processing resources and slows down the entire operation
  • It's seizure inducing
  • While FileZilla has a tabbed UI, updating of the displayed listings consumes so many resources that it may not even be possible to use other tabs during a recursive operation
To solve this dilemma, recursive operations will become a background task:
  • While FileZilla navigates the directory hierarchy it no longer displays the contents of each directory
  • A new status box below the remote file list will display the progress of the recursive operation. From there it is also possible to stop the operation. The location of the status box has been carefully chosen to avoid having to reposition other controls. For example a status box at the top of the remote pane would be incompatible with directory comparison.
  • It will be possible for the user to navigate the server during recursive operations.
What do you think?
recursive_operation.png
recursive_operation.png (82.57 KiB) Viewed 19304 times

User avatar
boco
Contributor
Posts: 24152
Joined: 2006-05-01 03:28
Location: Germany

Re: FileZilla 3 development diary

#992 Post by boco » 2015-06-22 15:23

Looks like an excellent improvement. Could you apply it to deleting recursively, too?
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Do yourself a favor and read Network Configuration.
All FileZilla products fully support IPv6. http://worldipv6launch.org
### END SIGNATURE BLOCK ###

User avatar
botg
Site Admin
Posts: 31574
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: FileZilla 3 development diary

#993 Post by botg » 2015-06-22 16:01

That's already the case in the repository ;)

User avatar
botg
Site Admin
Posts: 31574
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: FileZilla 3 development diary

#994 Post by botg » 2015-06-26 12:05

Finally: https://tools.ietf.org/html/rfc7568: "Deprecating Secure Sockets Layer Version 3.0"
SSLv3 MUST NOT be used.
This makes me very happy.

User avatar
boco
Contributor
Posts: 24152
Joined: 2006-05-01 03:28
Location: Germany

Re: FileZilla 3 development diary

#995 Post by boco » 2015-06-27 01:36

Should kick those slackers into gear, eventually.
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Do yourself a favor and read Network Configuration.
All FileZilla products fully support IPv6. http://worldipv6launch.org
### END SIGNATURE BLOCK ###

User avatar
botg
Site Admin
Posts: 31574
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: FileZilla 3 development diary

#996 Post by botg » 2015-07-09 14:59

OpenSSL 1.0.2d came out today: http://openssl.org/news/secadv_20150709.txt

Since it's a client-side vulnerability, no update for FileZilla Server is required.

User avatar
botg
Site Admin
Posts: 31574
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: FileZilla 3 development diary

#997 Post by botg » 2015-07-24 09:40

Proof-of-concept exploit for FTP over TLS data connection stealing

Full exploit here: https://filezilla-project.org/misc/ftps ... xploit.cpp

Below the description:

Preamble

FTP suffers from data connection stealing vulnerability [1][2]. While there are mitigations for plaintext FTP they are incomplete. Without bi-directional authentication, an attacker can steal the data connections and do tricks like swapping connections between users.

There are two mechanisms in FTP over TLS that can solve connection stealing:
- Client certificates [3][a] and
- TLS session resumption [4][5]

In the following we will discuss how TLS session resumption benefits FTP over TLS and how incorrect implementations, despite requiring session resumption on the data connection, still allow connection stealing.

FTP over TLS and TLS session resumption

Not only does session resumption speeds up the data connection handshake, it also guarantees the authenticity of the data connection: If the same session is used for both the control connection and the data connection, both client and server know that the data connection is authentic.

With an unresumed session, the client can only check that the connection comes from the server. The client however cannot verify that the data connection is associated with his control connection.

Likewise, without resumption the server cannot verify the authenticity of the client.


Session resumption can be facilitaded through different means:
- Session IDs and a server-side session cache
- Session tickets sent to the client

Both work equally well for FTP over TLS.


The Exploit

Due to an implementation flaw in some servers, an attacker can inject his own session into the server's TLS context.

The problem is that these servers, while requiring session resumption, do not verify that it is indeed the same session on both control- and data connection. Thus, if an attacker can inject a session, he can then resume it and thus successfully steal data connections.


Let's assume Alice wants to download a file from Bob. She has established a control connection with Bob and has just issued the PASV and RETR commands. Mallory sees Alice's attempt to establish the data connection and connects to the server in place of Alice. Mallory does however not know a valid session on the server.
After the data connection handshake, Bob checks whether the connection was resumed and, noticing it isn't, drops it.

At this point however, Mallory now knows of a valid session: The one that just got closed! He now either has a session ID or a session ticket.

The next time Alice wants to transfer a file using the same control connection to Bob, Mallory attempts to steal the connection again. Mallory presents his own session in the TLS handshake. Bob sees that the connection is resumed. Not checking that the session matches the session of the data connection, Bob then proceeds sending the file to who he believes is Alice.



In the following we assume that the FTP server uses OpenSSL to keep things simple and concrete. There is no reason however to assume that the problem does not occur with other TLS implementations.

When performing a handshake that isn't resumed, OpenSSL adds the new session to its session cache and sends a session ticket to the client. This is done before the FTP server checks whether the session is resumed or not, thus even if the server decides to drop the unresumed connection, the client now has valid session credentials.

The next time the client connects he can thus present a valid session to the server.


Solutions

There are at least two feasible solutions:

If the server sees an unresumed handshake on the data connection, the server should close not only the data connection but also the control connection [7]. Note that this requires that the lifetime of session IDs and session tickets is bound to the lifetime of the control connection. A server that uses a single long-lived TLS context cannot use this solution.

The server can verify that the session of the control connection is the same as the session of the data connection. Note: At least with OpenSSL, it is not obvious how to do this [6].


Affected servers

This is a partial list of affected servers, there may be many more that
haven't been tested.
- vsftpd 3.0.2 or earlier, fixed in 3.0.3
- ProFTPD 1.3.5 or earlier, fixed in 1.3.5a
- FileZilla Server 0.9.50 or earlier, fixed in 0.9.51


Remarks

[a] I believe that with client certificates an attacker could still swap data
connections belonging to the same user.


References

[1] https://web.archive.org/web/20000303212 ... sec_01.txt
[2] http://cr.yp.to/ftp/security.html
[3] http://scary.beasts.org/security/CESA-2008-002.html
[4] http://scarybeastsecurity.blogspot.de/2 ... eased.html
[5] http://vincent.bernat.im/en/blog/2011-s ... c5077.html
[6] https://filezilla-project.org/misc/session_matching.c
[7] Solution proposed by Chris Evans <scarybeasts@gmail.com>

User avatar
botg
Site Admin
Posts: 31574
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: FileZilla 3 development diary

#998 Post by botg » 2015-07-28 19:42

When import a queue containing 5 million files I noticed that FileZilla was consuming over 16 GiB of memory while the queue was being imported. That's far too much for so few files.

To process XML files, FileZilla so far has been using TinyXML. I have always known that TinyXML is not the best in terms of performance or memory footprint and furthermore TinyXML isn't being improved anymore so I started looking for an alternative.

Of course there was the option of TinyXML-2, the official TinyXML successor which is claimed to perform better. But how much better? What about other alternatives? After spending a bit of time using my favorite Internet search engine, I quickly I stumbled across pugixml which to my delight had a nice and recent benchmark, comparing many different XML libraries: http://pugixml.org/benchmark.html

Based on the benchmark data, two options suggested themselves to me: Either update to TinyXML-2 or switch to pugixml.

I had a quick look at the TinyXML-2 API documentation. Nothing spectacular there, very similar to the old TinyXML-1.

Next I studied the pugixml documentation and found a simple, well thought-out API. One of the key difference between TinyXML and pugixml is the way the DOM-tree is accessed. In TinyXML, the accessors to get child nodes return pointers. In pugixml they return handles and strings can never be null. Why does this matter? Well, look at this example:

Code: Select all

//TinyXML
std::string tinyxml_sample(TiXmlElement* foo) {
	if (foo) {
		auto bar = foo->FirstChildElement("bar");
		if (bar) {
			auto baz = bar->FirstChildElement("baz");
			if (baz) {
				auto t = baz->GetText();
				if (t) {
					return t;
				}
			}
		}
	}
	return std::string();
}

std::string pugixml_sample(pugi::xml_node foo) {
	return foo.child("bar").child_value("baz");
}
A striking difference! To be fair TinyXML has the TiXmlHandle class, but it has its own drawbacks, especially it cannot be used to modify the tree.

From here the way forward was clear: Migrate to pugixml. While tedious, changing FileZilla to use pugixml was a simple task. After fixing about two thousand compile errors, almost everything worked directly.

Memory usage importing that huge queue is down to about 40% and it only takes about 30% of the time to import the file.

Kuldip
500 Command not understood
Posts: 1
Joined: 2015-08-07 06:41
First name: Kuldip
Last name: Behal

Re: FileZilla 3 development diary

#999 Post by Kuldip » 2015-08-07 06:47

botg wrote:Some small quality of life changes are coming to FileZilla Server:
Connecting to server [::1]:14147...
Connected, waiting for authentication
Logged on
You appear to be behind a NAT router. Please configure the passive mode settings and forward a range of ports in your router.
Warning: FTP over TLS is not enabled, users cannot securely log in.
I want to make my windows 2008 R2 to be FTP server and using filezilla server for this but after the configuration I am getting the error:-
Warning: FTP over TLS is not enabled, users cannot securely log in.
You appear to be behind a NAT router. Please configure the passive mode settings and forward a range of ports in your router.

Please suggest the path.

User avatar
boco
Contributor
Posts: 24152
Joined: 2006-05-01 03:28
Location: Germany

Re: FileZilla 3 development diary

#1000 Post by boco » 2015-08-07 10:27

It's not an error, but a warning.

The TLS warning indicates you didn't create/import an SSL certificate and enable TLS, yet.
https://wiki.filezilla-project.org/SSL/TLS

The other is about the server not being reachable from the outside world as you are behind a NAT device (a router, usually). Please read the Network Configuration Guide, Passive chapter.

With those things configured, the warnings should disappear.
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Do yourself a favor and read Network Configuration.
All FileZilla products fully support IPv6. http://worldipv6launch.org
### END SIGNATURE BLOCK ###

User avatar
botg
Site Admin
Posts: 31574
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: FileZilla 3 development diary

#1001 Post by botg » 2015-10-06 20:37

Just a small teaser of some work in progress: libfilezilla. Expect more news in the coming days.

User avatar
botg
Site Admin
Posts: 31574
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: FileZilla 3 development diary

#1002 Post by botg » 2016-06-10 08:11

One feature I'm implementing for the next version of FileZilla are colored tabs.

As with many things, the feature itself is quite trivial. Adding color to a tab is easy. The complexity is integrating it with the surrounding features.

The biggest issue is how to integrate it into the Site Manager. As it goes, adding it to the site manager itself is also quite easy. Yet the complexity is once again integrating it with the surrounding features, in this case the reconnect and bookmark features, which indirectly use the site manager.

This complexity is the result of a dynamically grown feature set: The Site Manager has been implemented first. Later, the Bookmarks were added. Last but not least, tabs were implemented. Needless to say that no prior feature had been prepared in advance for the following feature.

To solve this mess I'm going to create a completely new model that will deal with the site data. All existing Site Manager and bookmarks functionality will be changed, or rewritten, to use this new model. With the new model, all users will hold references to the relevant site entry, instead of copying just some of the data or relying on getting it from elsewhere.

User avatar
botg
Site Admin
Posts: 31574
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: FileZilla 3 development diary

#1003 Post by botg » 2016-06-16 13:58

How does this look?
tabs.png
tabs.png (89.63 KiB) Viewed 14263 times

User avatar
boco
Contributor
Posts: 24152
Joined: 2006-05-01 03:28
Location: Germany

Re: FileZilla 3 development diary

#1004 Post by boco » 2016-06-19 01:34

Colorful. (Well, you had to ask! :P)
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Do yourself a favor and read Network Configuration.
All FileZilla products fully support IPv6. http://worldipv6launch.org
### END SIGNATURE BLOCK ###

User avatar
botg
Site Admin
Posts: 31574
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse
Contact:

Re: FileZilla 3 development diary

#1005 Post by botg » 2016-07-12 21:57

One of the upcoming changes I mentioned in viewtopic.php?t=36597 might soon come to FileZilla. I'm thinking of dropping support for non-UTF-8 encodings, or at least to severely reduce the fallback capabilities.

Maintaining the byzantine, borderline undefined fallback chains still in FileZilla becomes increasingly difficult, in particular since it's nigh impossible to test them, I don't have access to those broken servers what would require such a feature.

Is anybody reading this still using any non-UTF8 non-ASCII character encoding? I'd like to hear your rationale as to the why.

Post Reply