goaccess

> The other problem with logs is that it's very difficult to filter out all of the bot traffic. It's not very difficult, but isn't not effort-less. Start with something like https://github.com/allinurl/goaccess/blob/master/config/brow... which captures 99% of the crawlers out there. Then, when you notice there is one particular user-agent/IP/IP-range doing a bunch of requests, add it to list and re-run. Doing filtering based on ASNs that you see are being used for crawling lets you filter most of the AI agents too. We've been dealing with this problem for over 2 decades now, and there are solutions out there that removes almost all of it from your logs.

Goaccess (https://github.com/allinurl/goaccess) does this in real time and has been very stable for well over a decade.

Link if anyone else is curious (and lazy) : https://github.com/allinurl/goaccess EDIT: looked at the code... I don't see void* being passed around, data types are used, no excessive #ifdefs, it looks like a pretty clean C app to be honest. Maybe the problem is that OP is used to C++? (as the advice to use a special struct for strings would indicate) Either way, goaccess project looks awesome.

> C has type system TIL, apparently all that gobject code attempting to add something resembling a type system to C in glib is redundant... Edit: After taking a quick glance at some random listings in goaccess, I don't have the impression this was written by particularly talented C developers. Example: https://github.com/allinurl/goaccess/blob/master/src/sort.c#...

I'm waiting as well for the ability to use filters directly from goaccess. Hope they get to it soon! https://github.com/allinurl/goaccess/issues/117

I use GoAccess. It's an offline access.log analytics engine. One feature it has is to generate static site from its db. I have an hourly cron script that picks up the last hours logs and generates a static site. You can see it in action at https://www.clusin.com/analytics/ 1 - https://github.com/allinurl/goaccess

For us it was literately an `apt install goaccess`, picked the COMBINED log format and it's been running for over a month in prod without issues (rotating weekly). You should post the issue on their https://github.com/allinurl/goaccess, they may be able to help you.

Maybe not as lightweight, but GoAccess (https://github.com/allinurl/goaccess) does an awesome job at parsing the logs and displaying statistics, works for nginx and other webservers too

Very interesting. I just checked my logfiles and as expected most traffic seems to originate from search crawlers, feeds and bots running through all kinds of exploit urls. Hard to tell, but a wild guess is that more than 75% of hits and visitors are bot-related. I learned that it is possible to exclude bots through browsers.list [1] and in goaccess.conf you can exlude ip ranges. Unfortunately updating those entries is very time consuming and probably not worth it. [1] https://github.com/allinurl/goaccess/issues/560#issuecomment...

I like GoAccess for following reasons: * No javascript. * Does not need a database. * Realtime stats in html format. I use it in my blog, but also believe, that the numbers are completely inflated. I don't trust them. This has been discussed on Github a few times [1], so don't expect accurate numbers (yet). It's also hard to see what's going on recently on you server, because you only get totals. I'd love if I could change the time interval of the shown html stats. I like the way GoAccess is going though and I hope it will improve. [1] https://github.com/allinurl/goaccess/issues/964#issuecomment...

You can always create an issue on their github page, lots of help in there: https://github.com/allinurl/goaccess/issues

Goaccess works this way. https://github.com/allinurl/goaccess