splash

Interesting. I currently use https://github.com/scrapinghub/splash but will be interesting to see how this develops.

One of the supported projects is splash, which is basically WebKit-as-a-service. It takes an interesting approach to crawling where it renders the page using WebKit, and then exposes the "rendered DOM" -- so that your crawling code doesn't need to actually use JavaScript for information extraction. See: https://github.com/scrapinghub/splash People often use Scrapy + Splash together in the Python community for crawling more dynamic websites. A team I collaborated with is also working on a project to make Scrapy usable in a "cluster context", it's called scrapy-cluster. The idea is scrapy workers running across machines and a single crawling queue (in the current prototype, powered by Redis) in between them all. https://github.com/istresearch/scrapy-cluster

A shameless plug: https://github.com/scrapinghub/splash is an open source tool similar to Phantom JS, and it has an interactive web editor built-in. The concurrency model in Splash scripts is not the same as in Phantom JS scripts: Splash allows to use loops/ifs/functions in async code without recursion tricks or things like "casper.thenOpen". See http://splash.readthedocs.org/en/latest/scripting-tutorial.h... for more info. We haven't done a public announcement, and Splash is not yet feature complete, but it is used in production for more than a year, there are hundreds of unit tests and it is in active (though a bit sporadic) development - so give it a try if you want something different :) Installation is just "docker pull scrapinghub/splash" - see http://splash.readthedocs.org/en/latest/install.html.

Here is a similar project: https://github.com/scrapinghub/splash#renderpng