Ask HN: What is a simple tool for parsing and analyzing log files?

6 pointsthojest6y ago13 comments

Hi there, I have a small app (frontend, backend, database) with <100 users. I would like to use my log files to analyze basic metrics, like for example - number of requests - number of comments created - number of shared links per unit of time.

All of this runs on one single server, no big infrastructure, no big data. I am unable to find a SIMPLE tool to collect my logs in some way, parse them, and visualize them in histograms for example.

I know there exist ELK stack, Splunk, Graylog and many others, but all these solutions are much too complex. Especially I do not want to spend weeks setting this up correctly. Further more, most of these solutions need an extra server for aggregating the logdata in some timeseries db.

I would be very happy if you know about any opensource tool which can do this job.

13 comments

baccredited6y ago

I've had luck with https://goaccess.io/

example: goaccess logfile.log -o report.html --log-format=COMBINED

thojestOP6y ago

Now this looks really interesting and could be exactly what I was looking for. Many thx. Will check it out now.

thojestOP6y ago

this tool is amazing. tried it out and set it up yesterday evening :)

tekronis6y ago

Maybe checkout LNAV: http://lnav.org/

There's also angle-grinder, which has less features, but also pretty useful: https://github.com/rcoh/angle-grinder

gmuslera6y ago

Grafana´s Loki may have a lighter weight than the other examples you gave above.

For some kinds of logs there are tools for summarization and reports (like awstats for web or pflogsumm for mail servers).

And, of course, for particular queries on existing logs the standard text tools in a linux box let you generate a lot of info.

thojestOP6y ago

Grafana Loki was the one I really tried out. To be honest, I had the impression that this is still a bit alpha. You can work around a few shortcomings by reading from loki as a prometheus endpoint, but I experienced a few things in the data which were strange, and I could not observe when double checking the logs.

While it is true that the server footprint of promtail for collecting and pushig logs is much smaller, you still have to setup your loki sever for data aggregation. I spent nearly a week on the setup of promtail, loki and grafana and wasn't quite satisfied with stability and the end result. Of course this could be due to my limited experience considering log query language, time series db, prometheus, ... But overall I had the impression that what they aim for is quite similar to an optimized ELK stack.

runjake6y ago

For web logs, I still use Webalizer. For everything else, as long as we are not talking tens of gigs, I’ll be using some mix of Perl, Python, Shell, Awk, etc.

kjs36y ago

Syslog + something to massage the data (sh/awk/sed, perl, python)? Like we've been doing for simple apps for decades?

wdroz6y ago

It's even simpler today with systemd, you can write stuff like:

    journalctl -u nginx --since yesterday

kjs36y ago

That's not simpler than 'cat /var/log/messages' or the equivalent, if for no other reason than you still need something to digest/summarize the output. It's just different, mostly for the sake of being different, but I'll not bait the systemd fanbois any farther.

1 more reply

PaulHoule6y ago

pandas?

j / k navigate · click thread line to collapse

Ask HN: What is a simple tool for parsing and analyzing log files?

6 pointsthojest6y ago13 comments

All of this runs on one single server, no big infrastructure, no big data. I am unable to find a SIMPLE tool to collect my logs in some way, parse them, and visualize them in histograms for example.

I would be very happy if you know about any opensource tool which can do this job.

13 comments

baccredited6y ago

I've had luck with https://goaccess.io/

example: goaccess logfile.log -o report.html --log-format=COMBINED

thojestOP6y ago

Now this looks really interesting and could be exactly what I was looking for. Many thx. Will check it out now.

thojestOP6y ago

this tool is amazing. tried it out and set it up yesterday evening :)

tekronis6y ago

Maybe checkout LNAV: http://lnav.org/

There's also angle-grinder, which has less features, but also pretty useful: https://github.com/rcoh/angle-grinder

gmuslera6y ago

Grafana´s Loki may have a lighter weight than the other examples you gave above.

For some kinds of logs there are tools for summarization and reports (like awstats for web or pflogsumm for mail servers).

And, of course, for particular queries on existing logs the standard text tools in a linux box let you generate a lot of info.

thojestOP6y ago

runjake6y ago

For web logs, I still use Webalizer. For everything else, as long as we are not talking tens of gigs, I’ll be using some mix of Perl, Python, Shell, Awk, etc.

kjs36y ago

Syslog + something to massage the data (sh/awk/sed, perl, python)? Like we've been doing for simple apps for decades?

wdroz6y ago

It's even simpler today with systemd, you can write stuff like:

    journalctl -u nginx --since yesterday

kjs36y ago

1 more reply

PaulHoule6y ago

pandas?

j / k navigate · click thread line to collapse