ftr-site-config

This is exactly what Offpunk is doing: displaying the html page after it passed throught Readability. https://offpunk.net The whole page is still available with "view full" (or "v full") In the current trunk, if configured, it uses ftr-site-config rules to extract content for specific websites ( https://github.com/fivefilters/ftr-site-config ) I do 90% of my browsing using Offpunk (reading blogs and articles) and, suprizingly, it often works better than a graphical browser (no ads, no popup, no paywall). Of course, it doesn’t work when you really needs JS.

This is great! If it's useful, I work on a project where we maintain a repository of XPath selectors for extracting article content from many different sites: https://github.com/fivefilters/ftr-site-config - they're based on the original public Instapaper rules. We also have PDF generation, but it's not really for crawling, and wasn't created for reading on a device like the Supernote, more for printing and reading: https://pdf.fivefilters.org/simple-print/

Thanks for mentioning Instant View, I hadn't come across that. We actually maintain something similar here: https://github.com/fivefilters/ftr-site-config We use these in our own tools and also get contributions from others, including Wallabag users: https://github.com/wallabag/wallabag Before it was sold, Instapaper used to have something similar. A public database of its site-specific extraction templates. We used that as the starting point for our repository.

If you're trying to build one yourself, have a look at the open source Readability code[1]. It was originally developed by Arc90 and is now used by Apple and Mozilla in their browser reader views. The code has been ported to a number of different languages. I work on a service called Full-Text RSS[2] that used a PHP port of Readability, coupled with site-specific extraction rules[3] to identify and extract article content from each feed item. It then produces a full-text version of the given feed. The idea is you subscribe to the full-text version in whichever feed reader you use and it will give you full-text articles where you had partial content before. [1] https://github.com/mozilla/readability [2] https://www.fivefilters.org/full-text-rss/ [3] https://github.com/fivefilters/ftr-site-config

To help extract article content, you might be interested in this collection I help maintain: https://github.com/fivefilters/ftr-site-config/ It's used, in addition to an automatic article extractor, in Full-Text RSS: http://ftr.fivefilters.org

You might have come across this already, but we maintain a collection of article extraction rules for various sites here https://github.com/fivefilters/ftr-site-config - it was adapted from a database maintained by Instapaper in its early days and today has contributions mainly from users and developers of an open source Instapaper/Pocket alternative called Wallabag: https://github.com/wallabag/wallabag Also usable with a free version of Full-Text RSS available here: https://bitbucket.org/fivefilters/full-text-rss/src/master/

Something I've worked on: http://fivefilters.org/content-only/ Actually uses the original Insapaper site-specific extraction rules (they were hidden after Marco sold it I think) which can now be found here: https://github.com/fivefilters/ftr-site-config/ We get contributions from users of Wallabag (good open source alternative to Instapaper/Pocket). It also uses the original Arc90 Readability code ported to PHP to figure out where the article content is without any knowledge of the site. We sell the latest versions (AGPL licensed) but older versions go up in our public repository: https://bitbucket.org/fivefilters/full-text-rss

I thought you weren't getting the content text at all. :) Should be fixed now. We now strip that out: https://github.com/fivefilters/ftr-site-config/blob/master/e... But I don't consider that a big problem - lots of sites include things like that in their content. It's annoying, but no easy way to identify them all accurately and remove them without custom rules.

Can be broken down to roughly these steps: 1. The tool for extracting article content is Full-Text RSS[0]. It relies on arc90's original (open source) article detection algorithm, Readability, as well as a set of site-specific extraction rules which we maintain on Github[1]. 2. Our PDF Newspaper[2] application then cleans that extracted article output to normalise HTML/CSS styles so we can present the output in our own layout without things breaking. We use CSS for multi-column output. 3. Finally, we use Chromium's new headless API to request the generated output in step 2 and create three PDFs - injecting CSS to increase the font size each time. After we have the three PDF files, we check the PDFs in order of smallest to largest font size and as soon as a larger font size creates an extra new page, we discard that and keep the PDF we had

specific xpath are available here https://github.com/fivefilters/ftr-site-config