newspaper

You haven't give any details about programming language preferences but if you're interested in a Python approach then Newspaper3k is worth a look https://github.com/codelucas/newspaper

As others have noted, this doesn't seem to collect the full article text, just stuff that you would get from an RSS fee. From the title, I expected something more like newspaper3k[0]. I used that for an NLP class during undergrad to collect full-text news articles, in conjunction with Selenium (many mainstream sites don't work with just plain wget or requests). Lately I've starting using EpubPress[1] to grab full-text articles and generate an ePub, which happens every night via cron. Then I can get a full digest on my iPad over sftp at my leisure. Sadly EpubPress is not very sophisticated, sites like Bloomberg or ArsTechnica return "are you a robot" challenges which it can't bypass. I wish there was some kind of community driven library for retrieving full-text articles from common sites. In my vision of how that would work, users would c

Depending on what you're trying to do, there's also newspaper3k: https://github.com/codelucas/newspaper It's quite easy to get "good" extraction for large numbers of outlets/articles without a massive amount of special-casing, as news articles are nearly universally marked up with RDF metadata (partly for Google News's benefit). Article discovery, and perfect parsing, is quite a bit harder. I ended up rolling a new Scrapy project with site-specific parsing code for an academic project as I had quite specific requirements.

What makes this library different from something like Newspaper? https://github.com/codelucas/newspaper