gollum

The "OG" in the space is collectd, which is still my favorite choice if you are responsible at the operating system level: https://collectd.org/wiki/index.php/Chains https://github.com/elastic/logstash was one of the first modern approaches. I started using it less the more often I ran into JRuby related bugs. https://github.com/trivago/gollum is my pick from the golang ecosystem. There are many more variants depending on how much complexity you are trying to apply. If you need to apply machine learning models, for example, you're probably going to end up with something similar to Apache Storm, though I don't know if it's operational story has improved enough to consider it over other alternatives, I lost track years ago between Apache Spark and the half dozen other stream processing projects.

At trivago, we rely on the ELK stack to support our logging pipeline. We use a mix of filebeat and Gollum (github.com/trivago/gollum) to write logs into Kafka (usually we encode them using Protocol Buffers) these are later on read by Logstash and written into our ELK cluster. For us, ELK has scaled quite well both in terms of queries/s and writes/s and we ingest ~1.5TB of logs daily just in our on-premise infrastructure. The performance/monitoring team (where I work at) consists of only 4 people and although we take care of our ELK cluster it is not our only job (nor a component that requires constant attention).

We at trivago had a similar problem. For this we created Gollum: - http://tech.trivago.com/2015/06/22/gollum/ - https://github.com/trivago/gollum We use it heavily to stream any Kind of Data into Kafka: Access and errorlogs, Application Logs, etc. Did you consider it as well?