warcprox

Nice, I understand it is similar to ArchiveBox + its web extension. Now to be honest, while it's optimal to archive pages from you browser view I am not sure I want a random web extension to be in everything I see from a security point of view. I would rather have a local proxy doing it. Maybe something like the InternetArchive warcproc [0]. Haven't tried yet. - [0] https://github.com/internetarchive/warcprox

I've looked into archiving all the pages i visit as well and warcprox[1] has been bookmarked for a while now Hard drive storage space being so cheap in the ~$15/TB range makes this more feasible even for video archival [1] https://github.com/internetarchive/warcprox

Since this is pretty new, some background: WARC is a file format written and read by a rather small but specialized set of web crawler tools, most notably the internet archive's tooling. For example, its java-based crawler heretrix produces warc files. There are a couple of other very cool tools, such as warcprox, which can create web archives from web browser activity by acting as a proxy, pywb which play back the same to make archived versions browsable, and related libraries [1](shoutout to Noah Levitt, Ilya Kreymer and collaborators for building all this). The file format itself is an iso standard, and it's very simple: Capture all http headers sent and received, and all http bodies sent and received, and do simple de-duplication based on digests of the body. There is a companion format, CDX, which builds indexes from warc files (whic

Similar: github.com/InternetArchive/warcprox

Wow, it's really rare these days to see a tool that supports WARC. Despite being an ISO standard [1] and the default archive format of the internet archive, and despite a handfull of lovingly crafted tools (such as webrecorder [2], warcprox etc.), it never seems to have caught on in a broader context. Really a shame - I' deeply convinced that the ability to archive and replay requests is a technique for defending and strengthening user rights. Links: [1] https://www.iso.org/standard/44717.html [2] https://github.com/webrecorder/webrecorder-desktop [3] https://github.com/internetarchive/warcprox

As I've discovered in the past few weeks of archiving a soon-to-be offline service: archiving is tricky and often requires tampering with traffic. The IA uses a MITM proxy [0] to produce WARC files from intercepted traffic. Another software suite for producing high-fidelity archives injects JS [1] into pages served by its MITM proxy. [0] https://github.com/internetarchive/warcprox [1] https://github.com/webrecorder/wombat

There's no reason why a web browser bookmark action doesn't automatically create a WARC (web archive) format. Heck, with the cost of storage so low, recording every webpage you ever visit in searchable format is also very realistic. Imagine having the last 30 years of web browsing history saved on your local machine. This would especially be useful when in research mode and deep diving a topic. [1] https://github.com/machawk1/warcreate [2] https://github.com/machawk1/wail [3] https://github.com/internetarchive/warcprox EDIT: I forgot to mention https://github.com/webrecorder/webrecorder (the best general purpose web recorder application I have used during my previous research into archiving personal web usage)

I've enjoyed great success with various archiving proxies, including https://github.com/internetarchive/warcprox#readme and https://github.com/zaproxy/zaproxy#readme (which saves the content to an embedded database, and can be easier to work with than warc files). The benefit of those approaches over just save-as from the browser is that almost by definition the proxy will save all the components required to re-render the page, whereas save will only grab the parts it sees at that time. If it's a public page, you can submit the URL to the Internet Archive, and benefit both you and them

I've been thinking along those very same lines for a long time (this project makes me wish for more free time). I have half a mind to fork this and add something like https://github.com/internetarchive/warcprox, or at the very least walk through the generated HTML and brute-force inline all assets as a first pass :)

Yes – modern web archiving tools capture JavaScript, CSS, etc. The basic support where they just download URLs found in HTML has been around for years but the current best-practice is to use a real browser engine (e.g. PhantomJS, Firefox) which has either been integrated to store directly or simply is using a proxy like https://github.com/internetarchive/warcprox which adds every requested resource to a WARC file (https://en.wikipedia.org/wiki/Web_ARChive).

A WARC might have been put into the Wayback Machine; do a search there for the url. Failing that, there are proxies you can run locally that let you access the content of a WARC as if you were browsing the original site. https://github.com/internetarchive/warcprox is even by IA themselves, so clearly they're working on it. A viewer that let you browse the contents of a WARC the same way you can do for a zip would be really nice, but it's probably a separate project from redesigning the site. In fact, browsing zip files (and a bunch of other similar file types, of course) was added only a few months ago.