wayback

https://github.com/internetarchive/wayback/tree/master/wayba... https://akamhy.github.io/waybackpy/ https://wiki.archiveteam.org/index.php/Restoring

https://github.com/ArchiveTeam/grab-site might be helpful. I'm a fan of the ability to create WARC archives from a target, uploard the WARC files to object storage (whether that is IA, S3, Backblaze B2, etc), and then keep them in cold storage or serve them up via HTTPS or a torrent (mutable, preferred). The Internet Archive serves a torrent file for every item they host; one can do the same with WARC archives to enable a distributed archive. CDX indexes can be used for rapidly querying the underlying WARC archives. You might support cryptographically signing WARC archives; Wayback is particular about archive provenance and integrity, for example. https://www.loc.gov/preservation/digital/formats/fdd/fdd0005... ("CDX Internet Archive Index File") https://www.loc.gov/preservation/digital/formats/fdd/fdd0002... ("WARC, Web ARChive file forma

Try the archive.org CDX API, you might be able to find it. https://github.com/internetarchive/wayback/blob/master/wayba...

archive.org geocities scrapes go back to 1996, so it is plausible it could have survived: https://web.archive.org/cdx/search/cdx?url=geocities.com&mat... If you ever remember any of the details, the CDX API can probably help. https://github.com/internetarchive/wayback/blob/master/wayba...

Query the Internet Archive’s CDX server for this info. Consider caching the data for performance and good netizen reasons. https://github.com/internetarchive/wayback/blob/master/wayba...

I had a look into this as part of a research project. To reduce the burden of writing specific scraping software, I investigated the software listed by the Archive Team[0]. The command line tools and libraries aside (as they would require much more specific tailoring to make them work), I was particularly interested in HTTrack and Warrick. Warrick[1] is defined as a tool to recover lost websites using various online archives and caches. Warrick[2] requires some expert knowledge in order to get it up and running. I found Warrick was a bit outdated, so decided to try something similar but more up to date, and came across this (hartator/wayback_machine_downloader). I found this to be a bit easier to work with as it would allow me to download snapshots within a time period, which is what I needed for this project. After running it for ~12 hou

Which paid services are you referring to? It is likely that these services aren't distributing the projects they are based on, if so, then they are in compliance with the licenses of the open source projects, which probably don't require attribution unless you distribute them. This project started in 2015 btw. Another similar project called waybackpack started in 2016. There are probably more projects. IMO wayback-machine-downloader is the better project though. https://github.com/jsvine/waybackpack The Wayback CDX Server API these projects are based on is quite simple to use btw, just some JSON responses to decode. https://archive.org/help/wayback_api.php https://github.com/internetarchive/wayback/blob/master/wayba...

Making a better archiving tool would be a great open source project. Wayback machine's archiver appears to be open source: https://github.com/internetarchive/wayback

Thanks! We were writing up a response at the same time: The Wayback Machine data is stored in WARC or ARC files[0] which are written at web crawl time by the Heritrix crawler[1] (or other crawlers) and stored as regular files in the archive.org storage cluster. Playback is accomplished by binary searching a 2-level index of pointers into the WARC data. The second level of this index is a 20TB compressed sorted list of (url, date, pointer) tuples called CDX records[2]. The first level fits in core, and is a 13GB sorted list of every 3000th entry in the CDX index, with a pointer to larger CDX block. Index lookup works by binary searching the first level list stored in core, then HTTP range-request loading the appropriate second-level blocks from the CDX index. Finally, web page data is loaded by range-requesting WARC data pointed to by the

Their older CDX API provides that functionality: https://github.com/internetarchive/wayback/tree/master/wayba...

The index for Wayback is a massive sorted text file (called a CDX) containing a line for each URL and timestamp. For very large installations this index is sharded across multiple servers and queried in parallel. The lookups are done using plain old binary search. http://archive.org/web/researcher/cdx_file_format.php Each CDX record maps a URL-timestamp pair to a byte offset into an ARC or WARC file. These are essentially just gzipped HTTP responses concatenated together: http://archive.org/web/researcher/ArcFileFormat.php http://www.digitalpreservation.gov/formats/fdd/fdd000236.sht... The document is retrieved, uncompressed, URLs are rewritten, the navigation banner javascript injected and the result is sent to the client. The code is here: https://github.com/internetarchive/wayback