chancy

Chancy is an early-stage project to address a lot of the issues I've run into over the years trying to use Celery+Flower on small and large projects. It's now been used in a few projects with thousands of users, over a billion jobs, and hundreds of thousands of workflows. All Chancy features work with just Postgres, on 1 or 1000 workers or even embedded inside your existing asyncio servers like FastAPI. - Support for job priorities, retries, timeouts, scheduling, global rate limits, memory limits, unique jobs, cancellation, and more - asyncio-based worker with support for asyncio, threaded, process-based, and sub-interpreter job execution. - Configurable job retention for easy debugging and tracking - Minimal dependencies (only psycopg3 required) - Plugins for a dashboard, workflows, cron jobs, and more - Opti

This is a fairly old article, but this has become a very common pattern. For 90% of projects that just need a task queue for sending email or generating reports, you're not going to be doing millions of messages a second and anything that simplifies your stack is worth looking at. I've used these patterns so often to work around issues encountered with Celery that I've split them into their own framework, https://github.com/TkTech/chancy (feedback appreciated ;) There are lots of these, several of which are commercial ventures so there's definite interest behind it: https://worker.graphile.org/ (Node.js) https://riverqueue.com/ (Go) https://github.com/acaloiaro/neoq (Go) https://github.com/contribsys/faktory (Go) https://github.com/sorentwo/oban (Elixir) https://github.com/procrastinate-org/procrastinate (Python)