resque

Definitely, thanks for your feedback. Will be bringing up redis in the next post. Redis has some messaging functionality baked right in and can be a good solution as seen with https://github.com/defunkt/resque in Ruby as well. ZeroMQ is also an important technology to understand in the context of this discussion.

Work queues are a mostly solved problem, unless you've got unusual needs, like preposterous amounts of throughput. These guys don't have unusual needs, and they've got a paltry 16k tasks to keep track of. For that kind of job, use a library. You don't want to be the one who writes code to deal with restarting hung workers, or shunting pathological jobs to a retry-later queue, or changing pipeline topology live in production without a hiccup. I've written this stuff -- we had unusual needs -- so take it from me: you don't want to write this stuff. If you do most of your work in Python, Celery is pretty slick: http://celeryproject.org/ If you like Redis -- and who doesn't? -- then Resque or one of the alternate implementations will work well in Ruby or a plethora of other languages: https://github.com/defunkt/resque https://github.com/defun

I've got great news then: you can make the jump to RoR today! :) This news isn't about Rails implementing its own background queue, but rather creating a unified API for interacting with background queuing systems; of which there are many. Resque (crafted at GitHub [1]) is probably the most popular: https://github.com/defunkt/resque. 1: https://github.com/blog/542-introducing-resque

In short, Gearman is multi-language, multiple different backends, possible to do RPC-style synchronous processing. Resque and friends[1] are single-language, exclusively async, but also extremely transparent because they're just JSON blobs in redis. Resque has a huge following as well. [1]: https://github.com/defunkt/resque/wiki/alternate-implementat...

A quick tip would be to run everything process intensive in the backend, don't waste frontend resources with API calls, data processing etc, got something that takes more than 200ms to run, place it in a job que and do the processing later. Some tools for the job: https://github.com/defunkt/resque https://github.com/tobi/delayed_job Also don't forget to cache. You shouldn't worry too much if you do these two properly.

Looks a lot like GitHub's Resque(Ruby) https://github.com/defunkt/resque hopefully, this python variant will get all the good stuff that comes with Resque (like a Sinatra app to see your queues - very handy).

Check out Resque - http://github.com/defunkt/resque

I am currently using Resque(http://github.com/defunkt/resque) for a similar requirement. You might want to check it out. edit: This is in Ruby though.

While the site isn't open-source, they have open-sourced a ton of code. Just to name a few: http://github.com/github http://github.com/defunkt/resque http://github.com/mojombo/jekyll http://github.com/mojombo/god http://github.com/mojombo/bert http://github.com/mojombo/grit

Yes, it does. See the LPOPRPUSH and BLPOP protocol commands. There's a number of threads on the Redis Google Group about using Redis as a job queue. There's also Resque by a githubber. http://github.com/defunkt/resque

You probably wouldn't have to do that. Check into something like resque: http://github.com/defunkt/resque (depends upon your needs, of course)