fluent-bit

Some of the HN discussion about whether "new projects in C should be allowed" is moot: Fluent Bit was imported into git in 2015 [0] (a few months before Rust's first public release), and may be considerably older than that for all I know. I suppose incidents like this actually do give a reason to "rewrite it in Rust", when "it" is "widely deployed infrastructure written in C". OTOH, I'm sure there were plenty of non-memory-safety bugs introduced and later fixed over the years, and rewriting in Rust will recapitulate that subset of bugs. [0] https://github.com/fluent/fluent-bit/commit/49269c5ec3c74411...

> Libc is a lot more tricky about signals, since not all libc functions can be safely called from handlers. And this is a huge thing. People do all kinds of operations in signal handlers completely oblivious to the pitfalls. Pitfalls which often do not manifest, making it a great "it works for me" territory. I once raised a ticket on fluentbit[1] about it but they have abused signal handlers so thoroughly that I do not think they can mitigate the issue without a major rewriting of the signal and crash handling. [1] https://github.com/fluent/fluent-bit/issues/4836

Everything is handled in process, you can find some examples in the runtime test cases. [1] - https://github.com/fluent/fluent-bit/blob/master/tests/runti...

The first link in this article is about switching from fluentbit to fluentd. Me too. I have issue with fluenbit. Especially around their JSON processing. As in this issue: https://github.com/fluent/fluent-bit/issues/1588 If you want a simple log forwarding, fluentbit is really good. But if you found yourself started to tweak fluentbit config, or write custom plugin in C and (recompile yourself) then it's time to move to fluentd.

Last time I tried, fluent-bit did not support IPv6. Maybe it does now? (Maybe it does not: https://github.com/fluent/fluent-bit/issues/2056 has been opened recently). Anyway, compared to the solution in the article, it seemed way too fanciful. Fluentd would not just be slow, but also run out of memory. I am no ruby-head, but a then-colleague of mine helped configure it. It still ran out of memory. I had no patience for a log system that did not work out of the box on one computer basically just logging failed ssh login attempts.

I couldn't find a link to your git repo on the website btw (repo (https://github.com/fluent/fluent-bit) ). I see that your input/output plugins are written in C[0]. I'm guessing this is because of the constraints of the embedded environment, but it really doesn't seem like it would be worth it in a normal one. The LUA sandbox model (e.g. Heka) just seems highly preferable. My main problem with Logstash/Fluentd is precisely the fragility and non-robustness of the plugin system. [0] https://github.com/fluent/fluent-bit/blob/master/plugins/out...

For the last year we have been working in a lightweight log shipper product called Fluent Bit[0]. Originally made for Embedded Linux now is taking it place in common environments. It's pretty similar to Fluentd in architecture, some features are: - Event-Driven (async network I/O). - Input / Output plugins. - Data routing based on Tags. - Optional SSL/TLS for networking operations when required. Next major version 0.9 will come with buffering support (memory/file system). Ah, it's fully made in C. [0] http://fluentbit.io http://fluentbit.io/documentation/0.8 http://github.com/fluent/fluent-bit