quickscrape

the DOI issue is as described by "aroch" Quickscrape is capable of downloading PDFs, given a DOI I think: https://github.com/ContentMine/quickscrape If publishers listened to their readers we would have had 100% Open Access ten years ago. Traditional academic publishers do not listen and don't care. Even if I could convince say PLOS to do something about this it wouldn't change much. We need all or the majority of publishers to provide good embedded metadata. Not just an isolated one or two. I don't see a good mechanism for making that happen, sadly.

I've been developing the stack for The ContentMine (http://contentmine.org). In the next month or so, we will start scraping the entire scientific literature as it is published each day, and processing it through our 'fact extraction' pipeline. Done so far: - I've defined a JSON format for declarative web scrapers (ScraperJSON: https://github.com/ContentMine/scraperJSON) - made a Node library for web scraping with ScraperJSON scrapers (thresher: https://github.com/ContentMine/thresher) - as well as a command-line client (quickscrape: https://github.com/ContentMine/quickscrape) - and a small library of ScraperJSON scrapers for scientific publishers that is about to start expanding rapidly (https://github.com/ContentMine/journal-scrapers). Next step is to build the web interface that will let people compose data mining pipelines. Imagine so