librdkafka

Celery Flower TUI doesn't yet support Kafka in Python, though Kombu (the protocol wrapper for celery) does. It looks like Kaskade has non-kombu support for consumers and ? https://github.com/celery/celery/issues/7674#issuecomment-12... .. https://news.ycombinator.com/item?id=40842365 (Edit) Kaskade models.py, consumer.py https://github.com/sauljabin/kaskade/blob/main/kaskade/model... kombu/transport/confluentkafka.py: https://github.com/celery/kombu/blob/main/kombu/transport/co... confluent-kafka-python wraps librdkafka with binary wheels: https://github.com/confluentinc/confluent-kafka-python librdkafka: https://github.com/edenhill/librdkafka

the main repo is back: https://github.com/confluentinc/librdkafka No word from confluent at all on this yet.

You are right, but in practice that's not what happens. Companies do not rely on open source libraries, the developers working for such companies do. I can give you a realistic example. If you want to use Kafka and Go, your probably only option is to use https://github.com/confluentinc/confluent-kafka-go. Its LICENSE explicitly says "no warranty". Now, what if I find a bug in the library? Only two realistic solutions from my side: 1. I submit the issue and hope for the maintainers to fix it 2. I dig deeper and try to fix the issue. I submit the PR None of the above scenarios are guaranteed to have a happy ending. The issue could be ignored, or piled up among thousand of other (maybe higher prio) issues. My solution may not be optimal and could be rejected (or if it's optimal, nobody is taking a look at it, and it could remain open for wee

Can you clarify what you mean? AFAIK with manual commit you have the most control over when the commit happens Look at this blog post describing a data loss caused by auto-commit: https://newrelic.com/blog/best-practices/kafka-consumer-conf... Also there also may be more subtle issues with auto-commit: https://github.com/edenhill/librdkafka/issues/2782

I have found clients based on librdkafka [1] or sarama [2] to be pretty robust and capable. For SSL, librdkafka relies heavily on OpenSSL which can cause a few headaches IME. But the rest is straightforward. [1] https://github.com/edenhill/librdkafka [2] https://github.com/Shopify/sarama

I wonder if statistics provided by librdkafka (available also with confluent-kafka-go) could have been used to solve the issue with less effort. https://github.com/edenhill/librdkafka/blob/master/STATISTIC...

Have anyone using the C version in librdKafka and how was your experience? https://github.com/edenhill/librdkafka

Fairly straightforward overall using https://crates.io/crates/kafka. The client doesn't automatically handle Kafka node failures, so that's the responsibility of your application code. I haven't had the chance to try https://github.com/fede1024/rust-rdkafka yet, but it looks promising and partially wraps the C/C++ library https://github.com/edenhill/librdkafka.

This needs a [2015]. Back then there wasn't a single usable client for .net. Since then the non-Java clients have massively improved. In particular https://github.com/edenhill/librdkafka is fantastic. There's also https://github.com/dpkp/kafka-python/, with support for consumer groups and all the other modern features. The basic criticism of requiring a complex client is valid, however you cannot achieve the delivery guarantees that Kafka gives you without one. The alternative would be to have a local agent process like consul, but that wouldn't give you the throughput that Kafka gets. Disclaimer: I've built a C# client for Kafka based on librdkafa (https://github.com/ah-/rdkafka-dotnet), so I'm biased.