huey

> The Many Problems with Celery: — https://steve.dignam.xyz/2023/05/20/many-problems-with-celer... > The problems with (Python’s) Celery: — https://docs.hatchet.run/blog/problems-with-celery > Dramatiq motivation: — https://dramatiq.io/motivation.html Here are some alternatives: Dramatiq: https://github.com/Bogdanp/dramatiq RQ: https://github.com/rq/rq Huey: https://github.com/coleifer/huey Hatchet: https://github.com/hatchet-dev/hatchet

Most of the small python alternatives I've seen use Redis as backend: - https://github.com/rq/rq - https://github.com/coleifer/huey - https://github.com/resque/resque

I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge. At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is managed by Docker swarm, and all containers pull jobs from redis. For our scale, it works great. However, I don't have control over the resource utilization of individual steps, and in the past I've had issues with the pipeline blocking as a result of how I was chaining tasks together. I think something like Nextflow would remove these limitations, but one thing I think I would miss is the ability to debug individual pipeline steps locally with an interactive debugger. As far as I can tell, Nextflow has l

You probably dont need a message queue if you have redis. And quite a lot of code surrounding it. Which also makes it a message queue. Example: [0] What you might mean is that you might not need a complicated server setup e.g. Kafka for simple message queues? [0]:https://github.com/coleifer/huey/

I love Huey [1]. I switched everything over from Celery to it. It’s so simple it’s a godsend, even supports SQLite queue [1] https://github.com/coleifer/huey

You might also like huey (1). [1] https://github.com/coleifer/huey

Its right here bro, in all it's undeleted glory: https://github.com/coleifer/huey/issues/118

Thanks for giving me advice on database syncing, but I think that you kinda missed the point. It doesn't matter how I chose 10 seconds as a periodic task interval, problem was that to a simple question about feature support I got dismissive/borderline rude answer which made me lose confidence in Huey as a project and it's long-term maintainability. And it's not like I asked for something exotic, but a support for subminute periodic tasks which almost every other task queue has. That issue doesn't seem to be anywhere in Github, so it's deleted: https://github.com/coleifer/huey/issues?utf8=%E2%9C%93&q=is%... As you said, it's a free software and you can do whatever you want with it, but to me this is a huge red flag and I'm happy to not be part of it. ;)

I am a big fan of RQ along with Huey (https://github.com/coleifer/huey)

I was looking into python task queuing recently, here's some other choices from my notes: https://github.com/malthe/pq - postgres queue based on rq and ruby queue_classic https://github.com/coleifer/huey - sqlite, redis and in memory by coleifer (peewee creator). Possible to implement postgres storage layer simply. https://github.com/closeio/tasktiger - flexible redis-based python task queue alternative to celery https://dramatiq.io/ - actor based python job queue on redis/rabbitMQ https://github.com/GoogleCloudPlatform/psq - gcp pub/sub based task queue

Interesting that coleifer wrote the comment. I really respect their open source code. It’s no BS in their task cue program https://github.com/coleifer/huey

> These kinds of dismissive comments make me wonder if the writer ever created anything of substance in their life. Ad hominem [0]. And he has. I use the things he has built in production. He has contributed to Python community (and SQLite) to a great extent. Some of the amazing things he built [1], [2], [3]. [0] - https://yourlogicalfallacyis.com/ad-hominem [1] - https://github.com/coleifer/peewee [2] - https://github.com/coleifer/huey [3] - https://github.com/coleifer/sqlite-web

Why, my own of course! Peewee[1] is a lightweight ORM, and Huey[2] is a task queue. [1]: https://github.com/coleifer/peewee [2]: https://github.com/coleifer/huey