tasktiger

I've recently looked at tasktiger https://github.com/closeio/tasktiger. It's a simple queue system that helped me understand how workers and schedulers work.

Here's the list of my issues: https://github.com/closeio/tasktiger/issues?utf8=%E2%9C%93&q... It came down to I like the features of RabbitMQ: * RabbitMQ scales messages without needing tons of RAM * I don't have to decide between not persisting messages to disk with Redis vs only using half the machine's RAM [1] * Queue and task visibility is better in RabbitMQ * Support for purging all tasks in a queue * Tasktiger had lower throughput than Celery and Dramatiq, maybe needs lazy-forking? Things I did like about Tasktiger: * no feature bloat and it's possible to actually read it's source code[2]. [1]: To enable persisting to disk (Redis fork + save snapshotting) you must limit Redis to only using half the available RAM on a machine. [2]: Celery is split up into multiple convoluted, bloated, difficult to read repos.

Yep, TaskTiger also uses Lua quite extensively ([0]) to ensure atomicity when moving tasks between various stages of processing (queued, active, scheduled, errored). [0] https://github.com/closeio/tasktiger/blob/master/tasktiger/r...

> What exactly is fragile about this approach? We've supported scheduled tasks in TaskTiger (https://github.com/closeio/tasktiger) via Redis's sorted sets and haven't had any issues. Using sorted sets for scheduled tasks isn't the fragile part, gluing it all together to prevent losing jobs by shifting into backup lists (or hashes) is the fragile part in my experience.

> Things like scheduled jobs can be done through sorted sets and persistent jobs are possible by shifting jobs into backup queues, but it is all a bit fragile. What exactly is fragile about this approach? We've supported scheduled tasks in TaskTiger (https://github.com/closeio/tasktiger) via Redis's sorted sets and haven't had any issues.

We tried this at WakaTime and it didn't scale well. We also tried https://github.com/closeio/tasktiger but the only ones that work for us are based on RabbitMQ: Celery<4.0 https://github.com/Bogdanp/dramatiq

It was definitely fun to work on it :) That said, I think we've also seen quite a big positive ROI as well. It is true that we've spent quite some time working on TaskTiger where we could've used an off-the-shelf task queue system. However, since we've created it, it allowed us to develop new features a lot faster and spend a lot less time debugging issues. We were using Celery before and it was a big pain with memory leaks, periodic tasks not working as expected, some tasks being lost for no apparent reason, and poor visibility into both aggregate statistics and individual tasks. We spent too much time debugging Celery's issues instead of working on our core business. Here's some more context: https://github.com/closeio/tasktiger/issues/141#issuecomment... Advanced features like scheduled tasks, unique queues, and task locks have helped

Shameless plug: Our team at Close has been inspired by RQ when we created TaskTiger – https://github.com/closeio/tasktiger. We’ve been running it in production for a few years now, processing ~4M tasks a day and it’s been a wonderful tool to work with. Curious to hear what y’all think!

I was looking into python task queuing recently, here's some other choices from my notes: https://github.com/malthe/pq - postgres queue based on rq and ruby queue_classic https://github.com/coleifer/huey - sqlite, redis and in memory by coleifer (peewee creator). Possible to implement postgres storage layer simply. https://github.com/closeio/tasktiger - flexible redis-based python task queue alternative to celery https://dramatiq.io/ - actor based python job queue on redis/rabbitMQ https://github.com/GoogleCloudPlatform/psq - gcp pub/sub based task queue

If you liked Celery 3.x but don't like all the bugs in Celery 4.x, check out these alternative Python task queues: https://github.com/closeio/tasktiger (supports subqueues to prevent users bottlenecking each other) https://github.com/Bogdanp/dramatiq (high message throughput)

I figured I'd add a mention of the task queue we use at close.io: https://github.com/closeio/tasktiger We run several hundred tasktiger workers in production (deployed using Kubernetes) to process the various backend task queues for our service. Project is MIT-licensed if anyone is interested in another celery alternative.

Replaced it with Task Tiger (https://github.com/closeio/tasktiger)

See also tasktiger: https://github.com/closeio/tasktiger