ReadableWebProxy

Whee, a topic close to my heart! https://github.com/fake-name/ReadableWebProxy is a project of mine that started out as a simple rewriting proxy, but at this point is basically a self-contained archival system for entire websites, complete with preserved historical versions of scraped content. It has a distributed fetching frontend[1], uses chromium[2] to optionally deal with internet breaking bullshit (Helloooo cloudflare! Fuuuuucccckkkkk yyyyooouuuuuuu), supports multiple archival modes (raw, e.g. not rewritten and destyled, and a rewritten format which makes reading internet text content actually nice), and a bunch of other stuff. The links in fetched content are rewritten to point within the archiver, and if content is not already retreived, it's fetched on-the-fly as you browse. It also has plugin-based content rewriting features, al

https://github.com/fake-name/ReadableWebProxy Disclaimer - It's a project of mine, specifically written for my interests, and is very much not production ready. It does have a full blown spider in it, which supports distributed fetching and stuff via another project [1]. It does both rewritten archiving (basically overwriting the page style) and raw archiving. I also have a bunch of miscellaneous stuff like a custom python interface for chromium so I can handle jerberscript shitshows (cloudflare "protection" [2], reCaptcha [3], etc...). People seem really dead-set on ruining the internet. I keep meaning to implement WARC proxying to feed into the IA as a parallel output stream for the spider, but it's a lot of work and I have too many projects already. 1: https://github.com/fake-name/AutoTriever 2: https://github.com/fake-name/ChromeContr

Uh, what part is hard? It's literally 3 lines of work, if you use a tool that does most of the work for you. [Here](https://github.com/fake-name/ReadableWebProxy/blob/510cc41ae...) is a migration I have for one of my hobby projects. Tthe process is `alembic revision --autogenerate` Add two lines to the migration skeleton (4 if you care about downgrades). `alembic migrate +1`. I may have the alembic migration commands slightly wrong. It's been a while since I've done them without having bash scrollback handy.

Archiving backend: https://github.com/fake-name/ReadableWebProxy Stores the scraped content in a postgres database, with external blob storage for binary content. Additionally, historical records of the scraped content are kept, so I have something that acts kind of like the internet archive too.

How about Readable-Web Proxy? https://github.com/fake-name/ReadableWebProxy