JOjordansissel·Aug 27, 2013
I don't know what bottlenecks you had when were observing high resource usage in logstash, but, in general, if there's a performance problem, it is a bug, and we can fix it.
The next release of logstash (1.2.0 is in beta) has a 3.5x improvement in event throughput. For numbers: on my workstation at home (6 vcpu on virtualbox, host OS windows, 8gb ram, host cpu is FX-8150) - with logstash 1.1.13, I can process roughly 31,000 events/sec parsing apache logs. With logstash 1.2.0.beta1, I can process 102,000 events/sec.
Processing speed will vary greatly by what you are doing with your events and it doesn't make sense to generalize performance characteristics globally, especially with a metric that, alone, doesn't really tell me much (cpu utilization).
If it's slow, it's a bug. We can fix it. :)
Further, you can use hekad with logstash and wit
Read the rest
h elasticsearch (one or both together, it doesn't matter).
In terms of problems solved, logstash helps solve transport and real-time processing problems. In cases where the logstash agent is too resource intensive, the logstash community offers many alternatives on this site: http://cookbook.logstash.net/recipes/log-shippers/
The community (myself included) is very interested in helping logstash be a success for its users, so if you do see performance problems, things that behave weirdly, or anything strange, it's probably a bug, and we can fix it.
The short version of all of this is captured by the project principles listed in the logstash readme: https://github.com/logstash/logstash/#project-principles
<3