typhoeus

Don't forget Typhoeus. That's literally my Goto "nuke the API with 10,000 concurrent requests" library. It's also based on Curl so very stable. https://github.com/typhoeus/typhoeus hydra = Typhoeus::Hydra.new 10.times do request = Typhoeus::Request.new("www.example.com", followlocation: true) request.on_complete do |response| #do_something_with response end hydra.queue(request) end hydra.run

My first thought was the thread executor of Typhoeus [0] [0] https://github.com/typhoeus/typhoeus

I always see people using something like HTTParty or open-uri for pulling down the page. My preferred (by far) is typhoeus, as it supports parallel requests and wraps around libcurl. https://github.com/typhoeus/typhoeus

My recipe is to use Typhoeus (https://github.com/typhoeus/typhoeus) + Nokogiri. I have tried lots of different options including EventMachine with em-http-request and reactor loop and concurrent-ruby (both a re very poorly documented) Typhoeus has a built-in concurrency mechanism with callbacks with specified number of concurrent http requests. You just create a hydra object, create the first request object with URL and a callback (you have to check errors like 404 yourself) where you extract another URLs from the page and push them to hydra again with the same on another callback.