solid_queue

Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job (https://github.com/bensheldon/good_job). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly concerned how it performs in MySQL(https://github.com/rails/solid_queue/issues/567#issuecomment... , https://github.com/rails/solid_queue/issues/508#issuecomment...). They also still have no support for batch jobs: https://github.com/rails/solid_queue/pull/142 .

Under the hood, it uses good ol' fork and keeps track of the generated process IDs. It's surprisingly simple. You can check out the relevant source here: https://github.com/rails/solid_queue/blob/main/lib%2Fsolid_q...

I'm seeking alternatives for my current queue system, which uses Golang and Redis. I'm interested in something similar to Rails' Solid Queue (https://github.com/rails/solid_queue). During my search, I came across River (https://riverqueue.com). Has anyone here used River in a production environment? I'd appreciate any feedback or experiences.

There’s also the new built in SolidQueue. https://github.com/rails/solid_queue/

How is this technically different from what solid queue is doing on rails ? https://github.com/rails/solid_queue Just trying to understand. I do get that hatchet would be language agnostic, SDK API kind of a solution.

if you have very high volume, there can be some tricks to using an rdbms as a queue system. For Rails use, 37Signals recently released the open source solid queue package, meant for use with either msyql or postgres. It makes some odd choices, like separating things into different tables depending on state. And it also makes careful use of the not-necessarily-super-well-known (but standard, and in both mysql and postgres) `FOR UPDATE SKIP LOCKED` clause. These choices were to get good performance at high volumes, which they think they have done and tested. So it can be done! Probably. And indeed solid queue is MIT-licensed, there are not barriers to looking at the choices they made and copying them in whatever language/platform you want. But it's not necessarily trivial. At non-huge volumes though it's probably not an issue. https://git

Rails isn't super opinionated about database writes, its mostly left up to developers to discover that for relational DBs you do not want to be doing a bunch of small writes all at once. That said it specifically has tools to address this that started appearing a few years ago https://github.com/rails/rails/pull/35077 The way my team handles it is to stick Kafka in between whats generating the records (for us, a bunch of web scraping workers) and and a consumer that pulls off the Kafka queue and runs an insert when its internal buffer reaches around 50k rows. Rails is also looking to add some more direct background type work with https://github.com/basecamp/solid_queue but this is still very new - most larger Rails shops are going to be running a second system and a gem called Sidekiq that pulls jobs out of Redis. In terms of read queries