parser

Pretty cool! (I can't test it, seems unavailable in Germany). How are you handling the text extraction with Readability.js? Does that run in a WebView in the background? I've built a similar, albeit less feature-complete, read-it later app in the past [1]. Spent most of my rewriting the core of the Postlight parser [2] in native Swift. Anyway, cool project, and I'd probably buy this once its available. [1] https://franz.hamburg/writing/read-later-app.html [2] https://github.com/postlight/parser

Thoroughly scraping is challenging, especially in an environment where you don’t have (or want) a JavaScript runtime. For content extraction, I found the approach the Postlight library takes quite neat. It scores individual html nodes based on some heuristics (text length, link density, css classes). It the selects the nodes with the highest score. [1] I ported it to Swift for a personal read later app. [1] https://github.com/postlight/parser

I haven’t directly compared them, but I have also found mercury parser (https://github.com/postlight/mercury-parser) to be pretty reliable. Its advantage is that you can directly pass it a page instead of having to give it a DOM. Since it turns a website into very plain (X)HTML it‘s fairly easy to use it to make a browsing proxy or automatically produce epub files for e-readers, which is what I do. Edit: Here’s the proof of concept type code I use: https://gist.github.com/solarkraft/d6306f17a761fcb5ce47f2be7... It’s a bit crappy, but it works for me :-)

Good luck! Those HTML issues you're coming across are tough and so varied across the web! I was working with Mercury Parser (pluggable parsing for different sites) in the past. https://github.com/postlight/mercury-parser

I believe the Instant View[1] crowdsourcing model where people write templates for each website could boost a lot these parsers (hope they open source it soon). Its just impossible to make these extensions work for every single website with some simple heuristics. Check the codebase of some popular parsers: Firefox (already mentioned): https://github.com/mozilla/readability/blob/master/Readabili... Google Chrome: https://github.com/chromium/dom-distiller Mercury parser: https://github.com/postlight/mercury-parser [1] https://instantview.telegram.org/

Sadly not - I'd love it to do that, but the Pocket API doesn't make that available. I've been contemplating building an add-on for Dogsheep that can do this for any given URL (from Pocket or other sources) by shelling out to an archive script such as https://github.com/postlight/mercury-parser - I collected some suggestions for libraries to use here: https://twitter.com/simonw/status/1401656327869394945 That way you could save a URL using Pocket or browser bookmarks or Pinboard or anything else that I can extract saved URLs from an a separate script could then archive the full contents for you.

Nice job! I love the UI. For hackers who want to find the best open sourced tools to work/parse news data (disclaimer: I'm a creator to some of those): 1. PyGoogleNews [1] -- If Google News had a Python library 2. NewsCatcher [2] -- Programmatically collect normalized news from (almost) any website 3. NewsPaper3k [3] -- Article scraping & curation 4. Mercury Parser [4] -- Extracting content from chaos [1] https://github.com/kotartemiy/pygooglenews` [2] https://github.com/kotartemiy/newscatcher [3] https://github.com/codelucas/newspaper [4] https://github.com/postlight/mercury-parser

I've personally used mercury-parser (https://github.com/postlight/mercury-parser/) in the past. It can display an article in clear text, Markdown, and HTML. I go with Markdown in order to have a permanent copy of the article within my notes. Slightly related: I'm always confused when I stumble upon an article that I can view just fine in Firefox's reader view, but doesn't get recognised as an article by Mozilla's Pocket, making me unable to highlight stuff.

I personally use postlight/mercury-parser[0] to convert articles to Markdown files, a small script to add extracted metadata (like author, featured image, original link, date it was scraped) to the top of a Markdown file, and put those Markdown files within Hugo for a DYI Pocket alternative. You can use the --format flag to pick between Markdown/text/HTML output, so it should serve your purpose. If anyone here has looked for a reader view on Chrome, odds are you've probably stumbled upon Mercury Reader[1]. This is what powers it. [0] https://github.com/postlight/mercury-parser/ [1] https://chrome.google.com/webstore/detail/mercury-reader/okn...

The repository is here: https://github.com/postlight/mercury-parser If you want to try it out while the API is still accessible, visit https://mercurizer.herokuapp.com/