newspaper

Nice job! I love the UI. For hackers who want to find the best open sourced tools to work/parse news data (disclaimer: I'm a creator to some of those): 1. PyGoogleNews [1] -- If Google News had a Python library 2. NewsCatcher [2] -- Programmatically collect normalized news from (almost) any website 3. NewsPaper3k [3] -- Article scraping & curation 4. Mercury Parser [4] -- Extracting content from chaos [1] https://github.com/kotartemiy/pygooglenews` [2] https://github.com/kotartemiy/newscatcher [3] https://github.com/codelucas/newspaper [4] https://github.com/postlight/mercury-parser

You haven't give any details about programming language preferences but if you're interested in a Python approach then Newspaper3k is worth a look https://github.com/codelucas/newspaper

As others have noted, this doesn't seem to collect the full article text, just stuff that you would get from an RSS fee. From the title, I expected something more like newspaper3k[0]. I used that for an NLP class during undergrad to collect full-text news articles, in conjunction with Selenium (many mainstream sites don't work with just plain wget or requests). Lately I've starting using EpubPress[1] to grab full-text articles and generate an ePub, which happens every night via cron. Then I can get a full digest on my iPad over sftp at my leisure. Sadly EpubPress is not very sophisticated, sites like Bloomberg or ArsTechnica return "are you a robot" challenges which it can't bypass. I wish there was some kind of community driven library for retrieving full-text articles from common sites. In my vision of how that would work, users would c

Depending on what you're trying to do, there's also newspaper3k: https://github.com/codelucas/newspaper It's quite easy to get "good" extraction for large numbers of outlets/articles without a massive amount of special-casing, as news articles are nearly universally marked up with RDF metadata (partly for Google News's benefit). Article discovery, and perfect parsing, is quite a bit harder. I ended up rolling a new Scrapy project with site-specific parsing code for an academic project as I had quite specific requirements.

What makes this library different from something like Newspaper? https://github.com/codelucas/newspaper

> This can be abused by bad actors to get obvious or difficult-to-refute falsehoods into the public memetic consciousness by repeating them. This is exactly why I started the project also I know there are hit pieces that are coordinated between the news-sites. I wonder how much other news that is reported is also coordinated reporting. However I have no direct proof of this and I cannot start from trying to prove something that I suspect. First I need to collect and analyse data fairly. So far I've just used newspaper to do the hard work of scraping for me (https://github.com/codelucas/newspaper) some basic filtering techniques e.g. I discard everything that has a question mark at the end of it (Betteridge's law of headlines and an MySQL to store articles etc. Then I run the second script to do some basic string analysis. I am going to be

> I imagine it starts with "links -dump", but then there's using the title as the filename, The title tag may exceed the filename length limit, be the same for nested pages, or contain newlines that must be escaped. These might be helpful for your use case: "Newspaper3k: Article scraping & curation" https://github.com/codelucas/newspaper lazyNLP "Library to scrape and clean web pages to create massive datasets" https://github.com/chiphuyen/lazynlp/blob/master/README.md#s... scrapinghub/extruct https://github.com/scrapinghub/extruct > extruct is a library for extracting embedded metadata from HTML markup. > It also has a built-in HTTP server to test its output as JSON. > Currently, extruct supports: > - W3C's HTML Microdata > - embedded JSON-LD > - Microformat via mf2py > - Facebook's Open Graph > - (experimental) RDFa via rdflib

Interesting topic and well written .. I kicked the tires a few times with newspaper3k https://github.com/codelucas/newspaper/ it has nlp and summary methods that work pretty well, I think I'll peek under the hood to see how it's being done there. Curious to see if your method is an improvement, if so, hey they're both in python!

https://github.com/mozilla/readability (that powers Firefox's reading mode) https://github.com/buriy/python-readability https://github.com/goose3/goose3 https://github.com/codelucas/newspaper For a complete solution, you can also look at Wallabag, which can be self-hosted.

I've used this in the past https://github.com/codelucas/newspaper There is also a wrapper to expose it as a rest api. Looks like it is not maintained, but easy enough to write your own https://github.com/voidfiles/newspaper-delivery

Not sure what do you use to scrape, but I can recommend newspaper to extract text. I contributed a while ago and it was pretty solid. https://github.com/codelucas/newspaper

Newspaper[0] forked Goose and added a number of improvements. Haven't used it in anger however so can't comment on its quality. https://github.com/codelucas/newspaper

I love this! When learning to program, I wrote a clone of a basic java game by Notch: Mine: http://codelucas.com/pages/island-overlord.html His: https://mojang.com/notch/ld12/breaking/ The code: Mine: https://github.com/codelucas/islandoverlord His: https://mojang.com/notch/ld12/breaking/BreakingTheTower.zip I also "cloned" a well known python article extraction library (goose) on Github with my own api & new functionality and my repo ended up doing super well (even better than the original). https://github.com/codelucas/newspaper vs. https://github.com/grangier/python-goose