heritrix3

I've considered running my own IA (https://github.com/internetarchive/heritrix3), which only archives my chrome history, which is stored in a local sqlite db. I haven't had the time to figure out the details of how the pieces should be glued together.

Also: the Internet Archive's crawler (Heritrix) implements link extractors for HTML, XML, CSS, JS, PDF, Word document, and even Shockwave Flash. Heritrix extractor Javadocs: https://www.javadoc.io/doc/org.archive.heritrix/heritrix-mod... Source of the JavaScript extractor: https://github.com/internetarchive/heritrix3/blob/master/mod... A script is included for running the extractors from the command line: https://github.com/internetarchive/heritrix3/blob/master/dis...

By the way, the technical side of this is very interesting. If you look at the tools mentioned (the wayback machine, but also perma.cc and other archival solutions), almost all of them rely on a single semi-modern tech stack that produces WARCs (web archives - ISO - ISO 28500:2017 https://iipc.github.io/warc-specifications/specifications/wa...). The main crawler still seems to be heritrix3 (https://github.com/internetarchive/heritrix3), but there's a great little ecosystem with tools such as webrecorder and warcprox. Still, I've read through the code of these tools and am feeling that they are failing in the face of the modern web with single page apps, mobile phone apps and walled gardens. Even newer iterations with browser automation are getting increasingly throttled and blocked and excluded from walled gardens. Perhaps the time has co

The British Library's web archive[1] does similar archiving work, but limits itself to 'British' sites - in other words, sites with a .uk domain. I've had good dealings with the admins before, when I submitted some of my personal sites for inclusion in the archive. Interestingly, the British Library uses web crawlers based on the Internet Archive's Heritrix web crawler[2], which demonstrates how important IA's work is for many other archival organisations' work. [1] https://www.webarchive.org.uk/ [2] https://github.com/internetarchive/heritrix3

Thanks! We were writing up a response at the same time: The Wayback Machine data is stored in WARC or ARC files[0] which are written at web crawl time by the Heritrix crawler[1] (or other crawlers) and stored as regular files in the archive.org storage cluster. Playback is accomplished by binary searching a 2-level index of pointers into the WARC data. The second level of this index is a 20TB compressed sorted list of (url, date, pointer) tuples called CDX records[2]. The first level fits in core, and is a 13GB sorted list of every 3000th entry in the CDX index, with a pointer to larger CDX block. Index lookup works by binary searching the first level list stored in core, then HTTP range-request loading the appropriate second-level blocks from the CDX index. Finally, web page data is loaded by range-requesting WARC data pointed to by the