exq

pervisor: http://elixir-lang.org/getting-started/mix-otp/supervisor-an... OTP: http://erlang.org/doc/ If you need a durable jobs, retries with exponential backoffs, dynamically scheduled jobs in the future - that are all able to survive application restarts, then an externally backed queueing library such as Exq could be a good fit. If you are starting a brand new project, I would also take a look at Faktory. It provides language independent queueing system, which means this logic doesn't have to be implemented across different languages and can use a thin client such as faktory_worker_ex."

n it's own thread which doesn't have a clean tracking mechanism via Sidekiq, so the only option to retry happens by re-queuing the job. Your database connection pool has to be increase to match your concurrency level in most cases as well and the concurrency has to be closely watched to track resource usage by job type in multiple job queues to ensure that too many heavy jobs don't take over and prevent other things like emails or push notifications from getting out on time. That's with everything that's built in plus a few extensions. Now, compare that to how processes work with Elixir/Erlang. Processes are so small (0.5kb compared with at least 1mb / thread) that each process is created with a supervisor as a standard mode of operation. Each job has it's own personal supervisor that will see if an error happened and immediately restart the job (with backoff). No external dependency, re-queuing or special tracking needed (although there are plenty of options). Because the processes are so small, in most cases you won't have a single large background job, you'll have many smaller ones breaking the entire thing into smaller pieces with other supervisors so at the point of an error, only that piece will need to restart. Worrying about different job sizes or resource usage largely goes out the window as well because the BEAM's scheduler limits the execution time of every process ensuring that a single large job can't takeover a machine. All of the smaller ones will keep executing just as if the big one wasn't running. Because the entire thing is built for concurrency, you also don't have to worry about the database-connection-per-job as the connection will just be checked out to make a query and then checked right back in. Because the scheduler could change context at any point, allowing a single process to hold a connection for longer than the database transaction in progress doesn't make sense. If you really want to use the Sidekiq approach, there is a library for it available (https://github.com/akira/exq). But what you get out of the box with Elixir/BEAM is already pretty far ahead of what Sidekiq gives you.