fake-name

ReadableWebProxy

Python2016–2019

fake-name / ReadableWebProxy

ReadableWebProxy

Rewriting web proxy and archival tool. At this point, it just tries to download all the things.

Explore on GitHub Read Hacker News discussions

4Distinct people

11Mentions

0Explicit recommenders

204GitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

Rewriting web proxy and archival tool. At this point, it just tries to download all the things.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

fake-name/autotriever · linked in 4 of the same discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 1:49:46 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year11 dated mentions

Observed mentions: 2016-08-23 18:55:52 UTC to 2019-12-22 10:40:52 UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Comments
2016	2016-01-01 00:00:00 UTC	2017-01-01 00:00:00 UTC	1	1
2017	2017-01-01 00:00:00 UTC	2018-01-01 00:00:00 UTC	1	1
2018	2018-01-01 00:00:00 UTC	2019-01-01 00:00:00 UTC	5	5
2019	2019-01-01 00:00:00 UTC	2020-01-01 00:00:00 UTC	4	4

What Hacker News says

11 sources

fake-name·Dec 22, 2019

Obligatory bump for my project ReadableWebProxy (https://github.com/fake-name/ReadableWebProxy) that was originally intended to do this. At this point, it does a GIANT pile of additional things most of which are specific to my interests, but I think it might be at least marginally interesting to others. It does both full autonomous web-spidering of sites you specify, as well as synchronous rendering (You can browse other sites through it, with it rewriting all links to be internal links, and content for unknown sites fetched on-the-fly). I solve the javascript problem largely by removing all of it from the content I forward to the viewing client, though I do support remote sites that load their content through JS via headless chromium (I wrote a library for managing chrome that exposes the entire debugging protocol here: https://github.co

Read the rest

m/fake-name/ChromeController https://pypi.org/project/ChromeController/).

In Show HN: Local Node.js app to save everything you browse and serve it offline

Reelin·Dec 5, 2019

Most of that seems oriented at full, manual archival rather than automated (lightweight-ish) text caching for search operations. ArchiveBox (https://github.com/pirate/ArchiveBox) looks promising, although it isn't capturing the "live" browser feed. Readable-Web Proxy (https://github.com/fake-name/ReadableWebProxy) also looks interesting, but seems to be external as well. Maybe I just need to suck it up and roll my own.

In How I Switched to Plan 9

mwest·May 17, 2019

Back in the mists of time, I used to use wwwoffle proxy. It was great for low-latency links, but also had the benefit of keeping an offline archive of whatever you'd browsed. Project's still there, although not sure how well it does with the modern web. http://www.gedanken.org.uk/software/wwwoffle/ There are a bunch of more modern variations too: https://archivebox.io/ - "Your own personal internet archive" https://getpolarized.io/ (as seen on HN previously) https://github.com/kanishka-linux/reminiscence https://github.com/fake-name/ReadableWebProxy

In Internet Data Is Rotting

fake-name·Feb 19, 2019

I read almost exclusively on a x86 device: A Windows Tablet. This is almost entirely because Windows 8 has the best touch-screen browser available on any platform: Internet Explorer Metro Edition (from a UX perspective, the actual browser internals are horrible). I've also written all my own server-side componentry[1]. 1: https://github.com/fake-name/ReadableWebProxy

In Bookworm: A Simple, Focused eBook Reader

fake-name·Nov 23, 2018

Whee, a topic close to my heart! https://github.com/fake-name/ReadableWebProxy is a project of mine that started out as a simple rewriting proxy, but at this point is basically a self-contained archival system for entire websites, complete with preserved historical versions of scraped content. It has a distributed fetching frontend[1], uses chromium[2] to optionally deal with internet breaking bullshit (Helloooo cloudflare! Fuuuuucccckkkkk yyyyooouuuuuuu), supports multiple archival modes (raw, e.g. not rewritten and destyled, and a rewritten format which makes reading internet text content actually nice), and a bunch of other stuff. The links in fetched content are rewritten to point within the archiver, and if content is not already retreived, it's fetched on-the-fly as you browse. It also has plugin-based content rewriting features, al

Read the rest

lowing the complete reformatting of content on-the-fly, and functions as a backend to a bunch of other projects (I run a translated light-novel/web-novel tracker site, and it also does the RSS parsing for that). I've been occatonally meaning to add WARC forwarding to the frontend, and feed that into the internet archive, but the fetching frontend is old, creaky and brittle (it's some old code), and does a lot of esoteric stuff that would be hard to replicate. [1]: https://github.com/fake-name/AutoTriever [2]: https://github.com/fake-name/ChromeController

In Archiving web sites

fake-name·Oct 1, 2018

https://github.com/fake-name/ReadableWebProxy Disclaimer - It's a project of mine, specifically written for my interests, and is very much not production ready. It does have a full blown spider in it, which supports distributed fetching and stuff via another project [1]. It does both rewritten archiving (basically overwriting the page style) and raw archiving. I also have a bunch of miscellaneous stuff like a custom python interface for chromium so I can handle jerberscript shitshows (cloudflare "protection" [2], reCaptcha [3], etc...). People seem really dead-set on ruining the internet. I keep meaning to implement WARC proxying to feed into the IA as a parallel output stream for the spider, but it's a lot of work and I have too many projects already. 1: https://github.com/fake-name/AutoTriever 2: https://github.com/fake-name/ChromeContr

Read the rest

oller 3: https://github.com/fake-name/WebRequest

In Inside Wayback Machine, the internet’s time capsule

fake-name·Aug 30, 2018

Uh, what part is hard? It's literally 3 lines of work, if you use a tool that does most of the work for you. [Here](https://github.com/fake-name/ReadableWebProxy/blob/510cc41ae...) is a migration I have for one of my hobby projects. Tthe process is `alembic revision --autogenerate` Add two lines to the migration skeleton (4 if you care about downgrades). `alembic migrate +1`. I may have the alembic migration commands slightly wrong. It's been a while since I've done them without having bash scrollback handy.

In Postgres data types you should consider using

fake-name·Jun 5, 2018

Archiving backend: https://github.com/fake-name/ReadableWebProxy Stores the scraped content in a postgres database, with external blob storage for binary content. Additionally, historical records of the scraped content are kept, so I have something that acts kind of like the internet archive too.

In Observations running 2M headless sessions

bunnybender·Feb 5, 2018

How about Readable-Web Proxy? https://github.com/fake-name/ReadableWebProxy

In Show HN: Simple Print – Convert web articles into printable PDFs

fake-name·Jun 22, 2017

In Verizon is killing Tumblr’s fight for net neutrality

fake-name·Aug 23, 2016

I do a significant amount of scraping for hobby projects, albeit mostly open websites. As a result, I've gotten pretty good a circumventing rate-limiting and most other controls. I suspect I'm one of those bad people your parents tell you to avoid - by that I mean I completely ignore robots.txt. At this point, my architecture has settled on a distributed RPC system with a rotating swarm of clients. I use RabbitMQ for message passing middleware, SaltStack for automated VM provisioning, and python everywhere for everything else. Using some randomization, and a list of the top n user agents, I can randomly generate about ~800K unique but valid-looking UAs. Selenium+PhantomJS gets you through non-capcha cloudflare. Backing storage is Postgres. Database triggers do row versioning, and I wind up with what is basically a mini internet-archive of

Read the rest

my own, with periodic snapshots of a site over time. Additionally, I have a readability-like processing layer that re-writes the page content in hopes of making the resulting layout actually pleasant to read on, with pluggable rulesets that determine page element decomposition. At this point, I have a system that is, as far as I can tell, definitionally a botnet. The only things is I actually pay for the hosts. --- Scaling something like this up to high volume is really an interesting challenge. My hosts are physically distributed, and just maintaining the RabbitMQ socket links is hard. I've actually had to do some hacking on the RabbitMQ library to let it handle the various ways I've seen a socket get wedged, and I still have some reliability issues in the SaltStack-DigitalOcean interface where VM creation gets stuck in a infinite loop, leading to me bleeding all my hosts. I also had to implement my own message fragmentation on top of RabbitMQ, because literally no AMQP library I found could reliably handle large (>100K) messages without eventually wedging. There are other fun problems too, like the fact that I have a postgres database that's ~700 GB in size, which means you have to spend time considering your DB design and doing query optimization too. I apparently have big data problems in my bedroom (My home servers are in my bedroom closet). --- It's all on github, FWIW: Manager: https://github.com/fake-name/ReadableWebProxy Agent and salt scheduler: https://github.com/fake-name/AutoTriever

In Web Scraping in 2016