capybara

Ruby's Capybara encapsulates Selenium and waits until elements appear on the page (the default timeout is 2 seconds). So you can write simple sequential code like click_link('bar') expect(page).to have_content('baz') and it will work even if the baz element is injected into the page by an Ajax request to the server triggered by clicking on bar. I've been using it for many years but I didn't check how they implement it. Maybe a callback from a MutationObserver? https://developer.mozilla.org/en-US/docs/Web/API/MutationObs... Documentation at https://github.com/jnicklas/capybara#asynchronous-javascript...

For this, we use Cucumber[0] with Capybara[1], Poltergeist[2] and PhantomJS[3]. [0] https://cucumber.io/ [1] https://github.com/jnicklas/capybara [2] https://github.com/teampoltergeist/poltergeist [3] http://phantomjs.org/

I recommend having a look at capybara [0]. It is build on top of nokogiri, and is actually a tool to write acceptence tests. But it can also be used for web scraping: you can open websites, click on links, fill in forms, find elements on a page (via xpath or css), get their values, etc... I prefer it over nokogiri because of its nice DSL and good documentation [1]. It also can execute javascript, which sometimes is handy for scraping. I've spend a lot of time working on web scrapers for two of my projects, http://themescroller.com (dead) and http://www.remoteworknewsletter.com, and I think the holy grail is to build a rails app around your scraper. You can write your scrapers as libs, and then make them executable as rake tasks, or even cronjobs. And because its a rails app you can save all scraped data as actual models and have them pers