FileZilla 3 development diary

Moderator: Project members

Message
Author
User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#976 Post by botg » 2015-02-05 18:07

Ever since 3.10 came out, there has been an issue with the progress bar in the queue updating very sluggishly when using SFTP in combination with speed limits. It's a complex issue to fix: What broke the progress bar updates actually greatly increases SFTP performance when no limits are set. Fixing it so that the speed remains great if no limit is set and the progress bar updates quickly if a limit is set requires extensive infrastructure changes.

What essentially happens is this: Data is uploaded in chunks. Once the server has received the chunk it sends an acknowledgement back. FZ updates its transfer each time an acknowledgement arrives. These chunks are quite big. With speed limits set to a low value, it takes a long time to transfer a chunk and the progress is thus rarely updated. In 3.10 chunk size has been increased significantly to match the bandwidth-delay product of the underlying TCP connection.

To solve this, my idea is to dynamically adjust the chunk size based on the currently set speed limit and potentially also the current transfer speed (as on very slow network connections the progress update can be slow as well).

For this to work, the current speed limit needs to be known by fzsftp.exe which handles the SFTP connection. Currently fzsftp has no idea about the actual limit, it only knows its own remaining quota in the current timeslice. (FZ does speed limits with token buckets; There's one bucket per connection, new tokens every 250ms)

To inform fzsftp about the current limit in an efficient manner, my idea is to send a command to fzsftp each time the limit changes.

However, while there already is code in FileZilla to inform some of its parts of option changes, this code currently only works for the GUI thread, it cannot inform non-GUI threads. All FTP and SFTP processing is done in a worker thread though, hence I need to extend this code to work with the thread event loop.

The two main challenges in implementing this are a) to make it fast, if the limit never changes there must be zero overhead and to b) make it thread-safe in the context of event handlers that can get asynchronously deleted.

Another issue with the current code is that it calls the option changed callback once for each option changed. I want it to be called only once, with a list of changed options passed to the callback. An easy changed compared to the above, but still tedious to change the existing users.

So far trying to fix this one bug has already caused some fallback:
- Fixed a bug with the total transfer size being too small in the log after a successful SFTP upload
- Inefficiencies passing the current transfer status around in FileZilla
- Even if no limit was set, the token buckets were needlessly updated every 250ms
- The speed limit code wasn't even thread-safe

While not much code has been implemented to address the original bug yet, the hardest part is over: Conceptualizing a solution.

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#977 Post by botg » 2015-02-12 16:33

RTTI in any language, C++ included, is slow.

If you have a simple type hierarchy though where you can change the base class and only ever need to cast to leaf classes, there's a simple trick you can use.

Here's a simple example showing how a holder for arbitrary types can be used for dynamic dispatch.

Code: Select all

#include <iostream>

struct holder_base {
    virtual ~holder_base() {}

    virtual char const* derived_type() const = 0;
};

template<typename T>
struct holder final : holder_base {
    holder() {}

    static char const* type() {
        static const char t = 0;
        return &t;
    }

    virtual char const* derived_type() const {
        return type();
    }

    T t_;
};

template<typename T>
bool is_type(holder_base const& b) {
    return b.derived_type() == holder<T>::type();
}


struct A{};
struct B{};

void stuff(A const& a)
{
    std::cerr << "Called with A\n";
}

void stuff(B const& b)
{
    std::cerr << "Called with B\n";
}

template<typename T> void do_dispatch(holder_base const &h)
{
    if (is_type<A>(h)) {
        stuff(static_cast<holder<T> const&>(h).t_);
    }
}

void dispatch(holder_base const& h) {
    do_dispatch<A>(h);
    do_dispatch<B>(h);
}

int main() {
    holder<A> foo;
    holder<B> bar;
    dispatch(foo);
    dispatch(bar);
    return 0;
}

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#978 Post by botg » 2015-04-07 09:32

Some small quality of life changes are coming to FileZilla Server:
Connecting to server [::1]:14147...
Connected, waiting for authentication
Logged on
You appear to be behind a NAT router. Please configure the passive mode settings and forward a range of ports in your router.
Warning: FTP over TLS is not enabled, users cannot securely log in.

User avatar
boco
Contributor
Posts: 26899
Joined: 2006-05-01 03:28
Location: Germany

Re: FileZilla 3 development diary

#979 Post by boco » 2015-04-07 10:40

Maybe a button or Help menu entry that leads to the https://ftptest.net page would be nice.
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Please do yourself a favor and read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
### END SIGNATURE BLOCK ###

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#980 Post by botg » 2015-05-20 19:05

A few words on Logjam:

FileZilla Client sadly accepts too small DH primes, I'll release 3.11.0.1 later this week to disallow tiny primes.

FileZilla Server on the other hand doesn't seem to suffer from Logjam for the simple fact that it doesn't seem to support DHE at all. The OpenSSL API makes it extremely difficult to use something simple as Diffie-Hellman, without first speaking some complex incantations when Saturn stands in the fifth house during harvest moon. Knowledge of the incantations is only passed on by divine enlightenment, you need to know them in order to learn them.

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#981 Post by botg » 2015-05-20 19:19

On a related note, it annoys me endlessly that most online testers for TLS only seem to deal with HTTPS and not other protocols using TLS, e.g. FTPS or SMTPS.

Luckily the domains tlstest.org, tlstest.com and tlstest.net were still available. Now I've got another project to work on.

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#982 Post by botg » 2015-05-22 22:01

Why do consumer-grade firewalls have such utterly shit quality?

First there's AVG which causes random corruption of transferred files, now there's Kaspersky that's not only preventing the data connection from becoming established, but also injects fraudulent TLS certificates into connections.

Expect FZ 3.11.0.2 in a couple of days, it'll blacklist Kaspersky's fake certificates.



Please tell me if you know of any other third-party malware that injects fake TLS certificates.

User avatar
boco
Contributor
Posts: 26899
Joined: 2006-05-01 03:28
Location: Germany

Re: FileZilla 3 development diary

#983 Post by boco » 2015-05-25 20:23

Many antivirus scanners do that now. For example avast (to my knowledge it only filters HTTPS so far).
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Please do yourself a favor and read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
### END SIGNATURE BLOCK ###

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#984 Post by botg » 2015-05-25 20:53

Let's throw actual security out of the window in order to give the user a false sense of security.

Such a great idea. Nothing could possibly go wrong.

User avatar
boco
Contributor
Posts: 26899
Joined: 2006-05-01 03:28
Location: Germany

Re: FileZilla 3 development diary

#985 Post by boco » 2015-05-25 21:10

First thing I did was *facepalm*.

Second thing was to disable that "feature".
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Please do yourself a favor and read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
### END SIGNATURE BLOCK ###

ajuaristi
504 Command not implemented
Posts: 8
Joined: 2015-05-29 13:02

Re: FileZilla 3 development diary

#986 Post by ajuaristi » 2015-06-08 15:46

Hi all,

Recently I scheduled a new patch for FileZilla implementing a per-host mapping to PPK keys: viewtopic.php?f=3&t=37073

Following Tim Kosse's feedback, I've already done my first tweaks at the code, and want to keep you informed. This is how it looks now under Linux:

Image

It does not do anything for now, but that'll come later.

Feel free to drop your feedback.

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#987 Post by botg » 2015-06-08 17:42

Where can I type in the filename directly?

The password field isn't needed here, as the aim is public key authentication.

What about this?
keyfile.png
keyfile.png (62.76 KiB) Viewed 57474 times

ajuaristi
504 Command not implemented
Posts: 8
Joined: 2015-05-29 13:02

Re: FileZilla 3 development diary

#988 Post by ajuaristi » 2015-06-08 18:11

Hi,

It's a good idea, but if I did that, the public key list on "Edit -> Settings -> SFTP" would not be needed anymore. In my example screenshot, the "Key file" drop-down would be filled with the PPKs added by the user in that list, so that you can select one of those.

I recall, it could be done your way, but the section "Edit -> Settings -> SFTP" would no longer be needed.

What do you think?

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#989 Post by botg » 2015-06-08 20:58

ajuaristi wrote:It's a good idea, but if I did that, the public key list on "Edit -> Settings -> SFTP" would not be needed anymore.
The list of keys in the settings would be used if there is no specific key given. Very useful feature for most users that only have one SSH private key and use it on all their servers, that way they do not have to create Site Manager entries. Users having a single SSH private key, with servers having a potentially large list of trusted public keys is SSHs preferred mode of operation after all, as is evident from the design of the OpenSSH reference implementation.

The primary use-case for the site-specific key on the other hand would be for users that have a tremendous amount of private keys and furthermore connect to servers that artificially limit the amount of keys the client can present. A complete inversion of the intended design.


Use-cases aside and into the realm of implementation details: In FZ, sites and settings are different entities, there is no coupling. Having a site depend on a setting (ie. the configured list of keyfiles) would introduce all sorts of new error case to be handled. Just some of the examples that immediately come to my mind:
- What happens if changing a key from the keyfile list, should the Site Manager entry change?
- What happens if a Site Manager entry references a key that isn't stored in the settings?
- Two instances of FZ running, one uses the Site Manager to change the keyfile for a site, the other has the settings open to manage keyfiles. Click OK in both dialogs. The latter in particular has interesting consequences if one were attempting to solve it.

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla 3 development diary

#990 Post by botg » 2015-06-22 07:27

Preview of a new feature

There are a number of recursive operations in FileZilla, for example when adding a remote directory tree to the queue. Currently, during these operations FileZilla navigates the remote directory hierarchy, entering and displaying each directory it visits. It is something that has worked well for over 14 years now.

There are however a number of problems with this approach:
  • While FileZilla does its work, the user cannot do anything else in the current tab
  • Displaying each directory as it is visited is slow, it consumes a lot of valuable processing resources and slows down the entire operation
  • It's seizure inducing
  • While FileZilla has a tabbed UI, updating of the displayed listings consumes so many resources that it may not even be possible to use other tabs during a recursive operation
To solve this dilemma, recursive operations will become a background task:
  • While FileZilla navigates the directory hierarchy it no longer displays the contents of each directory
  • A new status box below the remote file list will display the progress of the recursive operation. From there it is also possible to stop the operation. The location of the status box has been carefully chosen to avoid having to reposition other controls. For example a status box at the top of the remote pane would be incompatible with directory comparison.
  • It will be possible for the user to navigate the server during recursive operations.
What do you think?
recursive_operation.png
recursive_operation.png (82.57 KiB) Viewed 41734 times

Post Reply