FileZilla Log Analyzer Now Released!

Need help with FileZilla Server? Something does not work as expected? In this forum you may find an answer.

Moderator: Project members

Message
Author
Stunner
226 Transfer OK
Posts: 63
Joined: 2009-03-31 01:19
First name: A
Last name: Dog

FileZilla Log Analyzer Now Released!

#1 Post by Stunner » 2010-01-27 04:29

FileZilla Log Analyzer is now available on GitHub: https://github.com/Stunner/FileZilla-Log-Analyzer.

That is the location from which FileZilla Log Analyzer should be used for all issue reporting and collaboration. I am going to eventually take it off of Source Forge to help achieve this goal.

Original message:
========================

Ey fellas, some of you may have seen me snooping around here for awhile promising that I was creating a log parser that would make all of our lives easier. Well I finally released it minutes ago on sourceforge.net. You can find it at this link: http://sourceforge.net/projects/fla

It is in alpha mode as there are a number of features I would like to add to it, not so much because it is buggy(although it may have a few). I have extensively tested it(partly the reason I took so long to release it) and I am confident that there are no major bugs in it. To run it you will have to use terminal/command-line as there currently is no GUI, and it will produce text files with the interpreted data. Be sure to read the README file before using it as there is a good amount of stuff the script can do.

So be sure to check it out, and feel free to give me feedback on it. And if you are interested in helping with the project please let me know as I am always looking for people to help. If you have trouble with it at all or have bugs to report please contact me(my contact info can be found in the README with the project). Anyways I hope this helps with managing your server and makes your life a little easier. Enjoy!

List of all release announcements:
1.03 alpha: http://forum.filezilla-project.org/view ... 383#p68383
1.03a alpha: http://forum.filezilla-project.org/view ... 201#p69201
1.10 alpha: http://forum.filezilla-project.org/view ... 896#p69896
Last edited by Stunner on 2014-05-12 07:43, edited 4 times in total.

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla Log Analyzer Now Released!

#2 Post by botg » 2010-01-27 06:46

Thanks, looks promising.

Stunner
226 Transfer OK
Posts: 63
Joined: 2009-03-31 01:19
First name: A
Last name: Dog

Re: FileZilla Log Analyzer Now Released!

#3 Post by Stunner » 2010-02-19 08:23

I have noticed that there are quite a few posts with portions of server logs displayed and I am assuming most of the posters of those log files don't bother changing the IP addresses in those logs before posting them on the forum. I just wanted to make people aware that the FileZilla Log Analyzer can also be used to mask sensitive information by recreating your log file with scrambled user names, files/folders, and IP addresses. So I would highly recommend using that feature rather than blatantly revealing your user's(and your own) IP addresses for everyone on the net to see. And as always, feedback is appreciated! :)

User avatar
boco
Contributor
Posts: 26899
Joined: 2006-05-01 03:28
Location: Germany

Re: FileZilla Log Analyzer Now Released!

#4 Post by boco » 2010-02-19 10:27

Please note that in most cases posting logs with obscured IPs makes them useless from a support perspective. If you change all occurrences of the IP to another, that's OK, as long as you maintain consistency and regard the difference LAN IP vs. WAN IP.
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Please do yourself a favor and read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
### END SIGNATURE BLOCK ###

Stunner
226 Transfer OK
Posts: 63
Joined: 2009-03-31 01:19
First name: A
Last name: Dog

Re: FileZilla Log Analyzer Now Released!

#5 Post by Stunner » 2010-02-19 10:46

What do you mean by the "difference [between] LAN IP vs. WAN IP"? And yes, it does maintain consistency. It works this way: Upon first encounter of a new IP address it creates a random IP address and stores it in a dictionary with the original authentic IP address being used as a key. So whenever the real IP address is encountered again the same "fake" IP address is used instead. The same goes for user names and files/folders.

User avatar
boco
Contributor
Posts: 26899
Joined: 2006-05-01 03:28
Location: Germany

Re: FileZilla Log Analyzer Now Released!

#6 Post by boco » 2010-02-19 13:05

What do you mean by the "difference [between] LAN IP vs. WAN IP"?
Let's say there's a 192.168.0.1 in the log. If now your tool randomizes this IP to, say 123.234.432.321, we would now incorrectly assume the IP is public, while in fact it's not. So you would destroy information. Of course there's the other way around, if 66.54.208.99 would be randomized to 10.58.209.79, then it suddenly would be a LAN IP, again giving us the wrong impression. (All IPs are examples).

So your tool MUST exclude the private IP ranges from any randomization.
  • Class A - 10/8
  • Class B - 172.16/12
  • Class C - 192.168/16
  • Link local - 169.254/16
  • Loopback - 127/8
  • Loopback IPv6 - ::1/128
  • Link local IPv6 - fe80::/10
  • Site Local Unicast IPv6 - fec0::/10
  • Unique Local Unicast IPv6 - fc00::/7
All IPs in (shorted) CIDR notification. The list may not be complete.

If the source IP is in these ranges, don't touch it. If the result of a randomization is in these ranges, take a new seed and repeat the operation.

Private addresses are not unique and can't be routed over the internet. Thus they don't need "protection".
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Please do yourself a favor and read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
### END SIGNATURE BLOCK ###

Stunner
226 Transfer OK
Posts: 63
Joined: 2009-03-31 01:19
First name: A
Last name: Dog

Re: FileZilla Log Analyzer Now Released!

#7 Post by Stunner » 2010-02-19 16:44

Ah ok, interesting. I will be sure to fix that in the next update. Where can I find a complete list of these IP addresses? And thanks for pointing this out to me.

User avatar
boco
Contributor
Posts: 26899
Joined: 2006-05-01 03:28
Location: Germany

Re: FileZilla Log Analyzer Now Released!

#8 Post by boco » 2010-02-19 17:14

I got some from http://en.wikipedia.org/wiki/Private_network, but the complete list should be at http://www.iana.org/numbers/ (can't get more official :wink:).

IPv4: http://tools.ietf.org/html/rfc3330
IPv6: http://tools.ietf.org/html/rfc5156
### BEGIN SIGNATURE BLOCK ###
No support requests per PM! You will NOT get any reply!!!
FTP connection problems? Please do yourself a favor and read Network Configuration.
FileZilla connection test: https://filezilla-project.org/conntest.php
### END SIGNATURE BLOCK ###

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla Log Analyzer Now Released!

#9 Post by botg » 2010-02-19 18:45

See also IP_address

Stunner
226 Transfer OK
Posts: 63
Joined: 2009-03-31 01:19
First name: A
Last name: Dog

Re: FileZilla Log Analyzer Now Released!

#10 Post by Stunner » 2010-08-19 10:44

So I am about to finally embark on this now that I have some free time and was wondering what I should do in the case of IPv6 as the logs from FileZilla Server are all in the IPv4 format(unless I am missing something). So I would only enforce those restrictions for IPv4 and completely ignore those for IPv6, correct? Thanks.

User avatar
botg
Site Admin
Posts: 35491
Joined: 2004-02-23 20:49
First name: Tim
Last name: Kosse

Re: FileZilla Log Analyzer Now Released!

#11 Post by botg » 2010-08-19 20:23

Right now FileZilla Server does not even support IPv6 at all.

Legs
504 Command not implemented
Posts: 11
Joined: 2010-08-23 13:20
First name: Le
Last name: gs

Re: FileZilla Log Analyzer Now Released!

#12 Post by Legs » 2010-08-23 13:24

uhm there any version with a GUI in the planning that dont involve manual typing of all commands?
:roll:

cant get this manual version to work (copied log1.log to the install dir where FLA.py is located)

D:\parcer\FileZilla Log Analyzer-alpha1.02>python FLA.py -d log1.log
Analyzing log1.log...
Traceback (most recent call last):
File "FLA.py", line 1726, in <module>
main(sys.argv)
File "FLA.py", line 1649, in main
(m,p,d,t,u,i,a) = FLAmodule.getLine(line)
File "D:\parcer\FileZilla Log Analyzer-alpha1.02\FLAmodule.py", line 114, in getLine
return _parse(line)
File "D:\parcer\FileZilla Log Analyzer-alpha1.02\FLAmodule.py", line 72, in _parse
u = _parseUsr(6,fline) #user
File "D:\parcer\FileZilla Log Analyzer-alpha1.02\FLAmodule.py", line 42, in _parseUsr
return ' '.join((fline[i-1],_parseUsr(i+1,fline)))
File "D:\parcer\FileZilla Log Analyzer-alpha1.02\FLAmodule.py", line 42, in _parseUsr
return ' '.join((fline[i-1],_parseUsr(i+1,fline)))
File "D:\parcer\FileZilla Log Analyzer-alpha1.02\FLAmodule.py", line 42, in _parseUsr
return ' '.join((fline[i-1],_parseUsr(i+1,fline)))
File "D:\parcer\FileZilla Log Analyzer-alpha1.02\FLAmodule.py", line 42, in _parseUsr
return ' '.join((fline[i-1],_parseUsr(i+1,fline)))
File "D:\parcer\FileZilla Log Analyzer-alpha1.02\FLAmodule.py", line 38, in _parseUsr
if fline[0] == '(': #if next element is ip address
IndexError: list index out of range

D:\parcer\FileZilla Log Analyzer-alpha1.02>

Stunner
226 Transfer OK
Posts: 63
Joined: 2009-03-31 01:19
First name: A
Last name: Dog

Re: FileZilla Log Analyzer Now Released!

#13 Post by Stunner » 2010-08-23 21:53

Legs wrote:uhm there any version with a GUI in the planning that dont involve manual typing of all commands?
:roll:

cant get this manual version to work (copied log1.log to the install dir where FLA.py is located)

D:\parcer\FileZilla Log Analyzer-alpha1.02>python FLA.py -d log1.log
Analyzing log1.log...
Traceback (most recent call last):
File "FLA.py", line 1726, in <module>
main(sys.argv)
File "FLA.py", line 1649, in main
(m,p,d,t,u,i,a) = FLAmodule.getLine(line)
File "D:\parcer\FileZilla Log Analyzer-alpha1.02\FLAmodule.py", line 114, in getLine
return _parse(line)
File "D:\parcer\FileZilla Log Analyzer-alpha1.02\FLAmodule.py", line 72, in _parse
u = _parseUsr(6,fline) #user
File "D:\parcer\FileZilla Log Analyzer-alpha1.02\FLAmodule.py", line 42, in _parseUsr
return ' '.join((fline[i-1],_parseUsr(i+1,fline)))
File "D:\parcer\FileZilla Log Analyzer-alpha1.02\FLAmodule.py", line 42, in _parseUsr
return ' '.join((fline[i-1],_parseUsr(i+1,fline)))
File "D:\parcer\FileZilla Log Analyzer-alpha1.02\FLAmodule.py", line 42, in _parseUsr
return ' '.join((fline[i-1],_parseUsr(i+1,fline)))
File "D:\parcer\FileZilla Log Analyzer-alpha1.02\FLAmodule.py", line 42, in _parseUsr
return ' '.join((fline[i-1],_parseUsr(i+1,fline)))
File "D:\parcer\FileZilla Log Analyzer-alpha1.02\FLAmodule.py", line 38, in _parseUsr
if fline[0] == '(': #if next element is ip address
IndexError: list index out of range

D:\parcer\FileZilla Log Analyzer-alpha1.02>


Unfortunately no current plans for a GUI version, but I have definitely been thinking about it :). What I need to implement is more functionality first, such as the ability for the analyzer to take logs piece by piece and append it to the summary as opposed to requiring everything to be provided all at once.

Thanks for bringing the problem to my attention, but I can't really pinpoint the problem without looking at the log file/running some tests of my own. If you wouldn't mind sending the log file you are using to me via email(my email address can be found in the README) that would be great. It is a parsing error so it might be a case that I am not checking for, or something you have inadvertently changed in the log file.

Legs
504 Command not implemented
Posts: 11
Joined: 2010-08-23 13:20
First name: Le
Last name: gs

Re: FileZilla Log Analyzer Now Released!

#14 Post by Legs » 2010-08-25 15:12

sure thing. I edited out some personal information (replaced them with bogus numbers) and tested it again and same problem as before so its not corrupt.
log attached
Attachments
log1.log
(52.32 KiB) Downloaded 1245 times

Stunner
226 Transfer OK
Posts: 63
Joined: 2009-03-31 01:19
First name: A
Last name: Dog

Re: FileZilla Log Analyzer Now Released!

#15 Post by Stunner » 2010-08-26 01:35

Legs wrote:sure thing. I edited out some personal information (replaced them with bogus numbers) and tested it again and same problem as before so its not corrupt.
log attached
Really appreciate it. I fixed your issue but stumbled upon another which I am going to attempt to fix before making another release. I will post on this thread when I release the new version(should be fairly soon).

Post Reply