journal-scrapers

I've been developing the stack for The ContentMine (http://contentmine.org). In the next month or so, we will start scraping the entire scientific literature as it is published each day, and processing it through our 'fact extraction' pipeline. Done so far: - I've defined a JSON format for declarative web scrapers (ScraperJSON: https://github.com/ContentMine/scraperJSON) - made a Node library for web scraping with ScraperJSON scrapers (thresher: https://github.com/ContentMine/thresher) - as well as a command-line client (quickscrape: https://github.com/ContentMine/quickscrape) - and a small library of ScraperJSON scrapers for scientific publishers that is about to start expanding rapidly (https://github.com/ContentMine/journal-scrapers). Next step is to build the web interface that will let people compose data mining pipelines. Imagine so

> I've been building a library of ScraperJSON definitions for academic publisher sites, and I've converged on some formats that work for a majority of publishers with no modification (because they silently follow undocumented standards like the HighWire metadata). We've got a growing community of volunteers who will keep the definitions up to date for hundreds or thousands of journals. If you also use our scraper definitions for your metadata you'll get all the publishers for free. My approach has been to use the Zotero translators, since their 200,000 users have been alright at responding to publisher site changes. Unfortunately they are trapped in the Firefox ecosystem until someone converts their translators to generic js. Then Zotero could be a downstream consumer of the same scrapers, but also maybe maintain them as well. zotero's sc