FileZilla 3 development diary
Moderator: Project members
Re: FileZilla 3 development diary
Perhaps, if somebody submits high-quality patches.
Re: FileZilla 3 development diary
This means that I need to do myself?
Re: FileZilla 3 development diary
No, it merely means that somebody other than myself has to do it.
Re: FileZilla 3 development diary
And you can pass my request to the developers?
Re: FileZilla 3 development diary
I am "the developers".
Re: FileZilla 3 development diary
But who can do it?botg wrote:No, it merely means that somebody other than myself has to do it.
Re: FileZilla 3 development diary
Any skilled developer can do it.
Re: FileZilla 3 development diary
I made myself, because I'm a developer too ![Smile :)](./images/smilies/icon_smile.gif)
Create at "C:\Program Files\FileZilla\resources\alias" file "path_alias.xml"
Add this code to RemoteListView.cpp
How to use:
Press "Copy URL(s) to Clipboard" in context menu at RemoteListView field only
Thanks.
![Smile :)](./images/smilies/icon_smile.gif)
Create at "C:\Program Files\FileZilla\resources\alias" file "path_alias.xml"
Code: Select all
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<FileZilla3>
<Config>
<Alias_count>2</Alias_count>
</Config>
<Path_alias1>
<FTP_Path>ftp://my@mysite.ru/domain/my/big-e.ru</FTP_Path>
<HTTP_Path>http://big-e.ru</HTTP_Path>
</Path_alias1>
<Path_alias2>
<FTP_Path>ftp://me.nhl.com@nhl.com</FTP_Path>
<HTTP_Path>http://egor.cznhl.com</HTTP_Path>
</Path_alias2>
</FileZilla3>
Code: Select all
#include "xmlfunctions.h"
....
bool ReplaceAliasData(wxString& sURL, const wxString& aliasPath)
{
wxString sPath_alias=_T("");;
wxString FTP_Path;
wxString HTTP_Path;
wxString sAlias_count;
wxFileName fn(wxGetApp().GetResourceDir() + aliasPath, _T("path_alias.xml"));
TiXmlElement* pDocument = GetXmlFile(fn.GetFullPath());
if (!pDocument)
return false;
TiXmlElement* pConfig = pDocument->FirstChildElement("Config");
if (pConfig)
{
sAlias_count = GetTextElement(pConfig, "Alias_count");
}
if (!sAlias_count)
return false;
int Alias_count=0;
Alias_count=wxAtoi(sAlias_count);
wxString ii;
TiXmlElement* pAlias;
for (int i = 1; i <= Alias_count; i++)
{
//ii.Format(wxT("%d"),i);
ii=wxString::Format(wxT("%i"),i);
sPath_alias=_T("Path_alias");
sPath_alias+=ii;
pAlias = pDocument->FirstChildElement(sPath_alias.mb_str());
if (pAlias)
{
FTP_Path = GetTextElement(pAlias, "FTP_Path");
HTTP_Path = GetTextElement(pAlias, "HTTP_Path");
if ((HTTP_Path!= _T("")) && (FTP_Path!= _T(""))) {sURL.Replace(FTP_Path, HTTP_Path);}
}
}
delete pDocument->GetDocument();
return pAlias != 0;
}
.....
url.Replace(_T(" "), _T("%20")); //<- this string exist
ReplaceAliasData(url, _T("alias/"));
....
urls.Replace(_T(" "), _T("%20")); //<- this string exist
ReplaceAliasData(urls, _T("alias/"));
Press "Copy URL(s) to Clipboard" in context menu at RemoteListView field only
Thanks.
-
- 500 Command not understood
- Posts: 1
- Joined: 2014-02-20 13:17
- First name: Lou
- Last name: Kay
Re: FileZilla 3 development diary
Excellent botg!
Security is mostly a superstition. It does not exist in nature, nor do the children of men as a whole experience it.
<Third-party URLs in signatures not allowed>
<Third-party URLs in signatures not allowed>
Re: FileZilla 3 development diary
I want to use some C++11 in FileZilla and FileZilla Server, but until recently I couldn't. Support for old operating systems is one reason, in particular Windows XP and OS X 10.4.
With FileZilla Server it was easy, only a single platform to support. I've upgraded the server's build infrastructure to use Visual Studio 2013. VS2013 doesn't support XP anymore without lots of tinkering. Did it once, it was painful and took ages. Then the OpenSSL bug happened, I dropped XP for good then.
With the client it's more complex. Easiest part is Linux, nothing to be done there, the infrastructure already supports most of C++11. Windows is harder, in particular since the official binaries are being cross-compiled on Debian using MinGW. Switching to Debian Jessie (aka testing) on the build system allowed me to switch to a far newer version of the MinGW cross-compiler. Cross-compiling the dependencies and then FileZilla using the new infrstructure worked without a glitch, I had expected far worse, MinGW used to be so problematic. Last on the agenda is OS X, the OS X binaries of FileZilla are built on an ancient Mac Mini [*] (isolated behind a non-consumer-grade firewall) running OS X 10.4. Apart from not supporting C++11, it's time to drop support for that unsupported operating system and turn over the box to a museum or something
I've already got a new Mac Mini running OS X 10.9, now I just need to set up the infrastructure on it.
Dropping support for OS X 10.4/10.5 leaves me slightly unhappy, it's after all the only big-endian system I have access too. It's good practice to write code that can handle either endianess, in particular since code parts not working with a different endianess often have other problems as well. This is particularly relevant as endian conversion often happens during I/O serialisation. Not doing that right can lead to security vulnerabilities.
[*]A lovely beige box it is. Why is it that over time, new white hardware inevitably turns yellow over time? Black on the other hand, not only does it look better, it also stays black over time.
With FileZilla Server it was easy, only a single platform to support. I've upgraded the server's build infrastructure to use Visual Studio 2013. VS2013 doesn't support XP anymore without lots of tinkering. Did it once, it was painful and took ages. Then the OpenSSL bug happened, I dropped XP for good then.
With the client it's more complex. Easiest part is Linux, nothing to be done there, the infrastructure already supports most of C++11. Windows is harder, in particular since the official binaries are being cross-compiled on Debian using MinGW. Switching to Debian Jessie (aka testing) on the build system allowed me to switch to a far newer version of the MinGW cross-compiler. Cross-compiling the dependencies and then FileZilla using the new infrstructure worked without a glitch, I had expected far worse, MinGW used to be so problematic. Last on the agenda is OS X, the OS X binaries of FileZilla are built on an ancient Mac Mini [*] (isolated behind a non-consumer-grade firewall) running OS X 10.4. Apart from not supporting C++11, it's time to drop support for that unsupported operating system and turn over the box to a museum or something
![Smile :)](./images/smilies/icon_smile.gif)
Dropping support for OS X 10.4/10.5 leaves me slightly unhappy, it's after all the only big-endian system I have access too. It's good practice to write code that can handle either endianess, in particular since code parts not working with a different endianess often have other problems as well. This is particularly relevant as endian conversion often happens during I/O serialisation. Not doing that right can lead to security vulnerabilities.
[*]A lovely beige box it is. Why is it that over time, new white hardware inevitably turns yellow over time? Black on the other hand, not only does it look better, it also stays black over time.
Re: FileZilla 3 development diary
Crap. OS X 10.9 is too new to build wxWidgets 2.8 on. Currently trying to somehow install the base SDK for OS X 10.6 on it, which is the newest version supported by wx 2.8.
Re: FileZilla 3 development diary
Done. I've managed to build install the SDK for OS X 10.6 on 10.9 and build binaries targetting 10.4. There's a great German expression for things like this: Von Hinten durch die Brust ins Auge.
Re: FileZilla 3 development diary
The future of OS X support
The good news first: OS X in general support is there to stay.
The bad: Minimum supported OS X version will be OS X 10.7, supporting older versions will result in a lot of ugliness.
Some data:
What does this mean?
This will mean that in the near future, roughly 15% of all OS X users will no longer receive FileZilla updates.
Apart from that, there's something interesting in the data: A small number of OS X users is emulating the PowerPC binaries on their x86 CPU, as OS X 10.5 is the latest version of OS X to ever support PPC.
The good news first: OS X in general support is there to stay.
The bad: Minimum supported OS X version will be OS X 10.7, supporting older versions will result in a lot of ugliness.
Some data:
Code: Select all
CPU OS X Share
x86 10.9 59.2%
x86 10.8 14.7%
x86 10.7 11.0%
x86 10.6 12.0%
ppc 10.6 0.8%
x86 10.5 1.5%
ppc 10.5 0.4%
OS X users out of all users: 11%
This will mean that in the near future, roughly 15% of all OS X users will no longer receive FileZilla updates.
Apart from that, there's something interesting in the data: A small number of OS X users is emulating the PowerPC binaries on their x86 CPU, as OS X 10.5 is the latest version of OS X to ever support PPC.
Re: FileZilla 3 development diary
Post-mortem of a bug in the auto-updater
On platforms without a package manager, a reliable auto-update mechanism is of immense importance to encourage the use of the latest software version. This is especially true in networked programs where security vulnerabilities can have vast consequences.
For years now FileZilla has had an auto-update mechanism. A version with a broken auto-update would mean that many, if not most users of FileZilla would remain unaware of a new version with potentially critical fixes.
As part of the release process for new FileZilla versions I always test the auto-update functionality in the new version. I did successfully test 3.9.0.3 when it got released and also tested 3.9.0.4 with success prior to its release. I was surprised to find out shortly after releasing the new version that the previous 3.9.0.3 was unable to download the update.
I quickly went to analyze the issue: Something wrong with a mirror? Something wrong in my network? Something wrong with FileZilla itself? Debug logs in combination with a packet dump created with Wireshark quickly showed that the problem was in FileZilla itself.
Knowing that the problem is with FileZilla, I then pulled the new version from the update mechanism to prevent additional harm.
From the logs and the dumps I had a rough idea during which part of the download process the bug would trigger, though I had to tediously step through the code with a debugger to actually narrow it down. Interestingly during debugging the bug would not always happen, indicating that this bug is sensitive to timing. This fact helped me a lot in finding the cause of the bug. I remembered that in 3.9.0.3 I had made a few changes to the network code to fix a rare crash on connection termination. This change introduced a regression, depending on how the connection was closed, when reusing the socket instance not all state from the previous connection was being reset. The fix for this regression was then simple: http://svn.filezilla-project.org/filezi ... threv=6062
Further analysis showed that the bug only affected HTTP redirect handling, something the primary download servers makes ample use of to spread the traffic across secondary mirrors. Incidentally the update test infrastructure didn't use HTTP redirects, which is reason why this bug slipped through unnoticed. Much additional testing on both the affected and the fixed versions did not reveal any more problems, so I pushed out 3.9.0.5.
With the bug fixed and the fixed version released, I still had to get the fix to the users of the affected users somehow as due to the bug they couldn't get the update. Fortunately the bug only affected redirects and the primary download servers are under my direct control. Since the auto-update tests in the past always were successful, I knew that there had to be some way to get 3.9.0.3 to update. Eventually I managed to change the server-side auto-update back-end, coupled with some crazy rewrite rules for Apache's mod_rewrite, to avoid using HTTP redirects, at the expense of additional traffic on the primary servers.
Conclusion
You can see the impact this workaround had on traffic consumption on this traffic graph from one of the servers: During September, the primary download servers had to handle around roughly 12 TiB of extra data, most of it during the first week after the 3.9.0.5 release. The graph also shows what looks like an exponential decline in 3.9.0.3 usage and you can clearly see the weekends. While not shown here, on hourly traffic graphs you could even see when office hours in Europe and the Americas begin and end.
In conclusion, I hope this post has provided at least some insight into one of the many challenges creating reliable auto-updaters and how to deal with unforeseen bugs in the auto-updater.
On platforms without a package manager, a reliable auto-update mechanism is of immense importance to encourage the use of the latest software version. This is especially true in networked programs where security vulnerabilities can have vast consequences.
For years now FileZilla has had an auto-update mechanism. A version with a broken auto-update would mean that many, if not most users of FileZilla would remain unaware of a new version with potentially critical fixes.
As part of the release process for new FileZilla versions I always test the auto-update functionality in the new version. I did successfully test 3.9.0.3 when it got released and also tested 3.9.0.4 with success prior to its release. I was surprised to find out shortly after releasing the new version that the previous 3.9.0.3 was unable to download the update.
I quickly went to analyze the issue: Something wrong with a mirror? Something wrong in my network? Something wrong with FileZilla itself? Debug logs in combination with a packet dump created with Wireshark quickly showed that the problem was in FileZilla itself.
Knowing that the problem is with FileZilla, I then pulled the new version from the update mechanism to prevent additional harm.
From the logs and the dumps I had a rough idea during which part of the download process the bug would trigger, though I had to tediously step through the code with a debugger to actually narrow it down. Interestingly during debugging the bug would not always happen, indicating that this bug is sensitive to timing. This fact helped me a lot in finding the cause of the bug. I remembered that in 3.9.0.3 I had made a few changes to the network code to fix a rare crash on connection termination. This change introduced a regression, depending on how the connection was closed, when reusing the socket instance not all state from the previous connection was being reset. The fix for this regression was then simple: http://svn.filezilla-project.org/filezi ... threv=6062
Further analysis showed that the bug only affected HTTP redirect handling, something the primary download servers makes ample use of to spread the traffic across secondary mirrors. Incidentally the update test infrastructure didn't use HTTP redirects, which is reason why this bug slipped through unnoticed. Much additional testing on both the affected and the fixed versions did not reveal any more problems, so I pushed out 3.9.0.5.
With the bug fixed and the fixed version released, I still had to get the fix to the users of the affected users somehow as due to the bug they couldn't get the update. Fortunately the bug only affected redirects and the primary download servers are under my direct control. Since the auto-update tests in the past always were successful, I knew that there had to be some way to get 3.9.0.3 to update. Eventually I managed to change the server-side auto-update back-end, coupled with some crazy rewrite rules for Apache's mod_rewrite, to avoid using HTTP redirects, at the expense of additional traffic on the primary servers.
Conclusion
You can see the impact this workaround had on traffic consumption on this traffic graph from one of the servers: During September, the primary download servers had to handle around roughly 12 TiB of extra data, most of it during the first week after the 3.9.0.5 release. The graph also shows what looks like an exponential decline in 3.9.0.3 usage and you can clearly see the weekends. While not shown here, on hourly traffic graphs you could even see when office hours in Europe and the Americas begin and end.
In conclusion, I hope this post has provided at least some insight into one of the many challenges creating reliable auto-updaters and how to deal with unforeseen bugs in the auto-updater.
Re: FileZilla 3 development diary
In the next version of FileZilla the following FTP over TLS changes will be made:
- Support for the insecure RC4 algorithm will be removed
- The certificate verification dialog will show subject alternative names
- When a TLS alert is received, it now says which alert got received
- Improved error reporting on underlying socket instead of the generic "error in the push/pull function"
- Mismatch of server hostname with the certificate's common name or subject alternate name will become an error.
- Session reuse on the data connection will be mandatory