neoq

This is a fairly old article, but this has become a very common pattern. For 90% of projects that just need a task queue for sending email or generating reports, you're not going to be doing millions of messages a second and anything that simplifies your stack is worth looking at. I've used these patterns so often to work around issues encountered with Celery that I've split them into their own framework, https://github.com/TkTech/chancy (feedback appreciated ;) There are lots of these, several of which are commercial ventures so there's definite interest behind it: https://worker.graphile.org/ (Node.js) https://riverqueue.com/ (Go) https://github.com/acaloiaro/neoq (Go) https://github.com/contribsys/faktory (Go) https://github.com/sorentwo/oban (Elixir) https://github.com/procrastinate-org/procrastinate (Python)

As a data point, there's a similar Go based project called Neoq: https://github.com/acaloiaro/neoq

Exciting time for distributed, transactional task queue projects built on the top of PostgreSQL! Here's the most heavily upvoted in the past 12 months Hatchet https://news.ycombinator.com/item?id=39643136 Inngest https://news.ycombinator.com/item?id=36403014 Windmill https://news.ycombinator.com/item?id=35920082 HN comments on Temporal.io https://github.com/temporalio https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Internally we rant about the complexity of the above projects vs using transactional job queues libs like: river https://news.ycombinator.com/item?id=38349716 neoq: [https://github.com/acaloiaro/neoq](https://github.com/acaloi... gue: [https://github.com/vgarvardt/gue](https://github.com/vgarvar... Deep inside can't wait to see some like ThePrimeTimeagen to review it ;) https://www.youtube.com/@ThePrimeTimeage

We are looking right now to use a stable PG job queue built in Go. We have found 2 already existing ones: * neoq: https://github.com/acaloiaro/neoq * gue: https://github.com/vgarvardt/gue Neoq is new and we found it to have some features (like scheduling tasks) that were attractive. The maintainer has also been responsive to fixing our bug reports and addressing our concerns as we try it out. Gue has been around for a while and is probably serving its users well. Looking forward to trying out River now. I do wonder if neoq and river might be better off joining forces.

I just want to commend OP - if they’re here - for choosing an int64 for job IDs, and MD5 for hashing the payload in Neoq, the job library linked [0] from the article. Especially given the emphasis on YAGNI, you don’t need a UUID primary key, and all of its problems they bring for B+trees (that thing RDBMS is built on), nor do you need the collision resistance of SHA256 - the odds of you creating a dupe job hash with MD5 are vanishingly small. As to the actual topic, it’s fine IFF you carefully monitor for accumulating dead tuples, and adjust auto-vacuum for that table as necessary. While not something you’d run into at the start, at a modest scale you may start to see issues. May. You may also opt to switch to Redis or something else before that point anyway. EDIT: if you choose ULID, UUIDv7, or some other k-sortable key, the problem isn’

Neoq (https://github.com/acaloiaro/neoq) is a background job processor for Go. It began from my desire to have a robust Postgres-backed job processor. What I quickly realized was that the interface in front of the queue was what was really important. This allowed me to add both in-memory and Redis (provided by asynq) backends behind the same interface. Which allows users to switch between different backends depending on their durability and performance constraints. E.g. in-memory for testing/development/low durability requirements, postgres when you're not running Google-scale jobs, and Redis for all the obvious use cases for a Redis-backed queue. This allows me to swap out job queue backends without changing a line of job processor code. I'm familiar with the theory that one shouldn't implement queues on Postgres, and to a large extent,

This is exactly the thesis behind neoq (https://github.com/acaloiaro/neoq). Match your queue infrastructure with your application infrastructure.