wayback-machine-downloader

I would like to be able to pull content out of the Wayback Machine with a proper API [1]. I'd even be willing to pay a combination of per-request and per-gigabyte fees to do it. But then I think about the Archive's special status as a non-profit library, and I'm not sure that offering paid API access (even just to cover costs) is compatible with the organization as it exists. [1] It looks like this might exist at some level, e.g. https://github.com/hartator/wayback-machine-downloader, but I've been trying to use this for a couple of weeks and every day I try I get a HTTP 5xx error or "connection refused."

I used wayback-machine-downloader, I think you need one of the forks to make it work though. https://github.com/hartator/wayback-machine-downloader

There is a fix for that. This might point the way: https://github.com/hartator/wayback-machine-downloader/issue...

Unless you are going to continue to run the site and have it change etc, there is no point doing this since archive.org already hosts static snapshots of sites. Depending on the site you would use different tools, for eg for MediWiki/DokuWiki sites you would import the latest database dump on archive.org. I have used wayback-machine-downloader before for completely static sites before: https://github.com/hartator/wayback-machine-downloader/

You can use wayback_machine_downloader to automate downloading the archived pages https://github.com/hartator/wayback-machine-downloader/

Specifically: https://wiki.archiveteam.org/index.php?title=Restoring which mentions https://github.com/hartator/wayback-machine-downloader and also this tip: > This is undocumented, but if you retrieve a page with id_ after the datecode, you will get the unmodified original document without all the Wayback scripts, header stuff, and link rewriting. This is useful when restoring one page at a time or when writing a tool to retrieve a site: > http://web.archive.org/web/20051001001126id_/http://www.arch... From the downloader's issues, you may or may not need to use this forked version if you encounter some errors: https://github.com/hartator/wayback-machine-downloader/issue... https://github.com/ShiftaDeband/wayback-machine-downloader

I just found this https://github.com/hartator/wayback-machine-downloader anyone have anything similarly interesting/cool/niche-useful ?

Send ArchiveTeam the link on IRC or here and we can save it to archive.org, then later you can use wayback-machine-downloader to grab it from archive.org. https://github.com/hartator/wayback-machine-downloader

Came here to say this. My guess is that Amazon paid them to go away. If my guess is accurate ( I could certainly be wrong ), then Amazon could have them add a robots.txt banning archive.org. If they do that access to the archive will be removed. Mirror it now if you want the content. One nice way to do so ( handy for any site that you think may vanish off Way Back Machine ): https://github.com/hartator/wayback-machine-downloader

There are a few out there ready to go. Here's one https://github.com/hartator/wayback-machine-downloader You just dump and sync to s3 and use terraform to provision the route53 and bucket setups. Yes they are mostly content sites. The hardest part is filtering adult domains assuming you don't want them. There are a staggering number of adult domains that expire every year and get huge traffic.

Some of it is on archive.org: https://web.archive.org/web/20140607001646/https://www.quirk... You can use this tool to download the files for the site: https://github.com/hartator/wayback-machine-downloader/

https://github.com/hartator/wayback-machine-downloader (I discuss it on my https://www.gwern.net/Search tutorial and use it every once in while eg to make my mirror of 'Climb Mount Improbable' https://www.gwern.net/docs/genetics/selection/www.mountimpro... or 'Hard Truths From Soft Cats' https://www.gwern.net/images/hardtruthsfromsoftcats.tumblr.c... )

They should be able to use https://github.com/hartator/wayback-machine-downloader and get at least a static version of the website back online.

Really cool, congrats! I have built something similar, but to retrieve a backup for one of my dead websites. It was a fun project. Shameless plug: https://github.com/hartator/wayback-machine-downloader/