good_job

Good Job is a strong attempt. I believe it's based around Advisory Locks though. https://github.com/bensheldon/good_job

Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job (https://github.com/bensheldon/good_job). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly concerned how it performs in MySQL(https://github.com/rails/solid_queue/issues/567#issuecomment... , https://github.com/rails/solid_queue/issues/508#issuecomment...). They also still have no support for batch jobs: https://github.com/rails/solid_queue/pull/142 .

Advisory locks aren’t all sunshine and rainbows. They can only be unlocked by the Postgres connection that acquired the lock. That means you need to track the connection, typically by dedicating a connection to the job that needs locking. Here’s a good issue describing the tradeoffs between a lock table and advisory locks. https://github.com/bensheldon/good_job/discussions/831

Good Job does the same for Rails https://github.com/bensheldon/good_job

In the world of Ruby, GoodJob [0] has been doing a _good job_ so far. [0] - https://github.com/bensheldon/good_job

For Rails apps, you can do this using the ActiveJob interface via https://github.com/bensheldon/good_job Had it in production for about a quarter and it’s worked well.

Idk about pgagent but any table is a resilient queue with the multiple locks available in pg along with some SELECT pg_advisory_lock or SELECT FOR UPDATE queries, and/or LISTEN/NOTIFY. Several bg job libs are built around native locking functionality > Relies upon Postgres integrity, session-level Advisory Locks to provide run-once safety and stay within the limits of schema.rb, and LISTEN/NOTIFY to reduce queuing latency. https://github.com/bensheldon/good_job > |> lock("FOR UPDATE SKIP LOCKED") https://github.com/sorentwo/oban/blob/8acfe4dcfb3e55bbf233aa...

Nothing wrong with using the DB as a mq, especially if the load is small enough. Plenty of tools are built on that, these two come to mind https://github.com/procrastinate-org/procrastinate https://github.com/bensheldon/good_job

Sidekiq Pro is great, we're paying for it! 10k a year I think. But for people who are interested in alternatives, I'd also suggest Good Job (runs on Postgresql). https://github.com/bensheldon/good_job

I'm the GoodJob author. Here's the class that is responsible for implementing Postgres's LISTEN/NOTIFY functionality in GoodJob: https://github.com/bensheldon/good_job/blob/10e9d9b714a668dc... That's heavily inspired by Rail's Action Cable (websockets) Adapter for Postgres, which is a bit simpler and easier to understand: https://github.com/rails/rails/blob/be287ac0d5000e667510faba... Briefly, it spins up a background thread with a dedicated database connection and makes a blocking Postgres LISTEN query until results are returned, and then it forwards the result to other subscribing objects.

My hobby project does ~1.5M jobs per day enqueued into Postgres, no sweat. I use https://github.com/bensheldon/good_job which uses PG's LISTEN/NOTIFY to lower worker poll latency.

I used a weird stack of: - Rails: so I have a UI quickly together with some admin and a interactive shell to allow me interact with the app easiser - GoodJobs:https://github.com/bensheldon/good_job Used this for any Rails based job - Postgres for everything. No Redis. I found redis is a source of pain when it misbehave and reach max mem with eviction policy. When Redis mem is big, restart it take quite a bit of time. - Huge server: I rent hetzner server and scale vertically. - Golang + sqlc: I used Rails SQL schema to pair it with golang sqlc https://sqlc.dev/ it makes write SQL way easier - Stimulus - Docker Compose for Rails deployment - SCP/Systemd for go-based servic https://mailwip.com is my email forwarding saas app.

I've been using Sidekiq for years, rock solid and no complaints. However, this recent addition to the space seems to be gaining traction and appears to have an excellent feature set - https://github.com/bensheldon/good_job