parsley

I wrote a language that's basically a superset of this (https://github.com/fizx/parsley/wiki) back in 2008 and used it to crawl a variety of insane job posting sites. As crawling complexity increases, pretty soon you want an actual programming language to specify things like crawl order and cache behavior. Multi-page behavior was very hard to describe declaratively for misbehaving sites. Also, it's a terrible default (for security reasons) to let the web pages you're parsing automagically initiate new requests to arbitrary urls. Such as it is, I believe that the following works in some version of parsley, though I doubt its an official release. { "articles": [ { "title": ".title a", "comment_link": "follow(.subtext a:nth-child(3) @href) .athing:nth-child(1) .default" } ] } At some point, these json

https://github.com/fizx/parsley/wiki looks pretty similar. Running this sort of thing as a service/api never panned out for us because you are almost universally robots.txt denied and/or blocked. We briefly tried, and supported a wiki of json extraction scripts at parselets.org, but it went nowhere after a few months.

Hah! tectonic and I applied to YC with almost exactly this in 2009?! We went as far as building a browser-based IDE-like environment for generating these, and a language called parsley for expressing the scrapes. If you're interested in this, you could check out some of our related open source libraries: Edit: I just open-sourced the scraping wiki site we created here: https://github.com/fizx/parselets_com http://selectorgadget.com https://github.com/fizx/parsley https://github.com/fizx/parsley-ruby https://github.com/fizx/pyparsley https://github.com/fizx/csvget > cat hn.let { "headlines":[{ "title": ".title a", "link": ".title a @href", "comments": "match(.subtext a:nth-child(3), '\\d+')", "user": ".subtext a:nth-child(2)", "score": "match(.subtext span, '\\d+')", "time": "

Parsley is great for this. http://github.com/fizx/parsley For Python: http://github.com/fizx/pyparsley There use to be a website for sharing "parselet" scripts.

The real magic behind this gem is another one called Parsley, which is an astounding library for screen scraping. http://github.com/fizx/parsley/tree/master They've even built an entire app on top of the library: http://parselets.com/

http://github.com/fizx/parsley/tree/master is a C library which represents the core of the parsing functionality. A PHP binding is quite possible, and indeed, there already are bindings for Python and Ruby. In fact, I might just go look at how PHP bindings are done in general...