logstash

Per ESA-2021-31 [1] the common mitigation is not sufficient for logstash: > The widespread flag -Dlog4j2.formatMsgNoLookups=true is NOT sufficient to mitigate the vulnerability in Logstash in all cases, as Logstash uses Log4j in a way where the flag has no effect. It is therefore necessary to remove the JndiLookup class from the log4j2 core jar, with the following command: Logstash 7.16.1 should be out today to fix this... update even if mitigated: > Users should upgrade to Logstash 6.8.21 or 7.16.1 once they are released (expected Monday 13th December). These releases will replace vulnerable versions of Log4j with Log4j 2.15.0. EDIT: 7.16.1 is out in GitHub, but not yet everywhere on elastic co: https://github.com/elastic/logstash/releases/tag/v7.16.1 [1] https://discuss.elastic.co/t/apache-log4j2-remote-code-execu...

Is ISO8601 without punctuation still ISO8601? Most log parsers I have seen would not pick up the Parker format. ex gr https://docs.python.org/3/library/datetime.html#datetime.dat... https://github.com/elastic/logstash/blob/v1.4.2/patterns/gro...

The "OG" in the space is collectd, which is still my favorite choice if you are responsible at the operating system level: https://collectd.org/wiki/index.php/Chains https://github.com/elastic/logstash was one of the first modern approaches. I started using it less the more often I ran into JRuby related bugs. https://github.com/trivago/gollum is my pick from the golang ecosystem. There are many more variants depending on how much complexity you are trying to apply. If you need to apply machine learning models, for example, you're probably going to end up with something similar to Apache Storm, though I don't know if it's operational story has improved enough to consider it over other alternatives, I lost track years ago between Apache Spark and the half dozen other stream processing projects.

I was excited and hopeful to try the ELK stack a while back, but, like all the other comments here, have decided it was too unreliable and brittle. I have an open issue on Logstash, spent a fair amount of time detailing it, but have gotten no feedback. And then I realized there's 600+ open issues! https://github.com/elastic/logstash/issues/4389 I'm considering using Syslog-ng and would love to hear if anyone has comments on that. Based on other comments here, will be checking out Riemann and Fluentd as well. https://syslog-ng.org/

They delete the logs after 2 days (free plan) and 60 days (most expensive plan available). The 100GB is only monthly transfer. Would be nice to have a cloud log service with unlimited retention that's searchable, but for now self-hosting is the best choice using logstash [1] for logging and statsd [2] for metrics. [1] https://github.com/elastic/logstash [2] https://github.com/etsy/statsd

There are also some serious unresolved issues in the current versions of the ELK stack bits, such as https://github.com/elastic/kibana/issues/5170, https://github.com/elastic/kibana/issues/1961, https://github.com/elastic/logstash/issues/3440. Even with all that though, it's still an incredibly useful tool for log analysis. It's a much different scenario than what one would use statsd, graphite, etc. for.

I don't know what bottlenecks you had when were observing high resource usage in logstash, but, in general, if there's a performance problem, it is a bug, and we can fix it. The next release of logstash (1.2.0 is in beta) has a 3.5x improvement in event throughput. For numbers: on my workstation at home (6 vcpu on virtualbox, host OS windows, 8gb ram, host cpu is FX-8150) - with logstash 1.1.13, I can process roughly 31,000 events/sec parsing apache logs. With logstash 1.2.0.beta1, I can process 102,000 events/sec. Processing speed will vary greatly by what you are doing with your events and it doesn't make sense to generalize performance characteristics globally, especially with a metric that, alone, doesn't really tell me much (cpu utilization). If it's slow, it's a bug. We can fix it. :) Further, you can use hekad with logstash and wit

Take a look at logstash [1], it's on GitHub [2]. I think it could replace or integrate with RedisLogHandler. logstash takes logs from various inputs (syslog, files, Redis, HTTP), filters/normalized the formats into JSON, and outputs various formats (ElasticSearch, Redis, MongoDB, Graylog2). There is a WebUI with search and graphs. It's designed to scale-out and run on multiple machines. [1] http://logstash.net/ [2] https://github.com/logstash/logstash