scrapely

Their ML approach is most likely based on their https://github.com/scrapy/scrapely project, which uses instance-based learning (as I understand it, lightweight ML) to scrape other pages from a few examples.

I agree, this is definitely a solved problem! If you need to build a solid web scraping stack which is going to be maintained by many people and is critical to your business, you have two options… to use Scrapy or to build something yourself. Scrapy has been tried and tested over 6-7 years of community development, as well as being the base infrastructure for a number of >$1B businesses. Not only this, but there is a suite of tools which you have been built around it – Portia for one, but also other lots of useful open source libraries: http://scrapinghub.com/opensource/). Right now most people still have the issue of having to use xpath or css selectors to run your crawl or get the data, but not for too long. There's more and more ways of skipping this step and getting at data automatically: https://github.com/redapple/parslepy/wiki/Use-

Wrapper induction library is separated from Scrapy: https://github.com/scrapy/scrapely. It is used in Portia under the hood. Portia can be seen as a tool to annotate scrapely templates and define crawling rules and post-processing rules. I'm not a Portia developer/user myself, but I think it is possible to get script code from Portia; it exports Scrapy spider to some folder. But I don't really know what I'm talking about, it is better to ask at https://groups.google.com/forum/#!forum/portia-scraper or at stackoverflow (use tag 'Portia').

Maybe you should give a try at Portia (http://scrapinghub.com/portia/). It does exactly what you mean. You may also be interested in this library: https://github.com/scrapy/scrapely

Are you actually using scrapy[1] or scrapely[2]? [1] http://scrapy.org/ [2] https://github.com/scrapy/scrapely

Portia uses https://github.com/scrapy/scrapely library for data extraction. It doesn't use XPaths for learning. There are some links to papers in scrapely README; scrapely is largely based on ideas from these papers, but there are many improvements. In short - yes, this is taken care of.