kombu

Celery Flower TUI doesn't yet support Kafka in Python, though Kombu (the protocol wrapper for celery) does. It looks like Kaskade has non-kombu support for consumers and ? https://github.com/celery/celery/issues/7674#issuecomment-12... .. https://news.ycombinator.com/item?id=40842365 (Edit) Kaskade models.py, consumer.py https://github.com/sauljabin/kaskade/blob/main/kaskade/model... kombu/transport/confluentkafka.py: https://github.com/celery/kombu/blob/main/kombu/transport/co... confluent-kafka-python wraps librdkafka with binary wheels: https://github.com/confluentinc/confluent-kafka-python librdkafka: https://github.com/edenhill/librdkafka

Where AWS is the likely migration path for an app if it needs to be scaled beyond dev containers, already having tested with an SQS workalike prevents rework. The celery Backends and Brokers docs compare SQS and RabbitMQ AMQP: https://docs.celeryq.dev/en/stable/getting-started/backends-... Celery's flower utility doesn't work with SQS or GCP's {Cloud Tasks, Cloud Pub/Sub, Firebase Cloud Messaging FWIU} but does work with AMQP, which is a reliable messaging protocol. RabbitMQ is backed by mnesia, an Erlang/OTP library for distributed Durable data storage. Mnesia: https://en.wikipedia.org/wiki/Mnesia SQLite is written in C and has lots of tests because aerospace IIUC. There are many extensions of SQLite; rqlite, cr-sqlite, postlite, electricsql, sqledge, and also WASM: sqlite-wasm, sqlite-wasm-http celery/kombu > Transport brokers support

I have found Celery to be a major pain. It's very complicated (Are your Celery tasks guaranteed at-most-once or at-least-once delivery? Are you sure?). There are bugs with the Redis broker that cause it to drop tasks: https://github.com/celery/kombu/issues/305 I heartily recommend: https://dramatiq.io/motivation.html - "if you’ve ever had to use Celery in anger, Dramatiq could be the tool for you".

Oh wow, I was so close to embarking on a Python task queue the other day because Celery was driving me nuts. I just wanted various apps/processes to be able to fire and listen for events, something that AMQP with a topic exchange seemed great for[1]. The thing that really tripped me up was the overlap in terminology between the AMQP protocol and Celery. Sometimes a queue means a queue, but sometimes it means an exchange. How do task names and routing keys interact? Why does the celery worker die when you listen to more than 3 queues on python 3.6 [2]? Don't get me wrong, for basic stuff celery is great, albeit with a bit of a learning curve for the beginner. (And who knows, maybe I'm just missing something when it comes to more complex applications.) I also think the devs have done a great job and are super responsive. I think they also m