mousehole

There was a lovely project by _why called mousehole, a little ruby proxy you’d run locally that would add a little overlay where you could comment and read comments on anything you browsed. The code is archived here, not sure if anyone is still doing anything derived from it. https://github.com/nogweii/mousehole

Another project in a similar vein was _why's mousehole proxy https://github.com/evaryont/mousehole

One similar project was _why's mousehole: https://github.com/evaryont/mousehole and hoodwink.d https://github.com/robbyrussell/hoodwinkd

I've done almost exactly this in the past. There's a hell of a lot of fiddling in keeping the xpaths both stable and general enough to be useful. One approach I found absolutely vital was to have a rewriting, caching proxy between the crawler and the upstream site. This proxy allowed me to rewrite the page content into something much simpler for the crawler to get to grips with (RSS or Atom, say). I used Celerity (http://celerity.rubyforge.org/) with a hacked-on Mechanize API to do the rewriting, which let me handle JS-heavy pages almost as easily as static HTML ones. My original inspiration for this was _why's Mousehole (the source for which is here: https://github.com/evaryont/mousehole, I've got no idea if it runs on recent Rubies). The proxy also gives you somewhere to raise an alert if, all of a sudden, your scraping fails becaus