HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
webrecorder
pywb
Python2016–2020
webrecorder / pywb

pywb

Core Python Web Archiving Toolkit for replay and recording of web archives

8Distinct people
10Mentions
0Explicit recommenders
1.7KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Core Python Web Archiving Toolkit for replay and recording of web archives

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year10 dated mentions

Observed mentions: 2016-07-09 17:12:35 UTC to 2020-07-31 05:08:17 UTC.

0123452016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments20162017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments20182019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 5 mentions; 0 submissions, 5 comments2020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2020

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC101
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC101
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC202
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC505
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC101

What Hacker News says

10 sources
It depends how deep you want to go into the web archiving rabbit-hole, but using something like `archivenow`[0] would probably be the local optimum in terms of convenience, robustness, and permanence. It's "a tool to push web resources into web archives", which basically entails making requests of archiving services (like the Wayback machine, for example). Using `archivenow --all <url>`, you can back up <url> to a whole bunch of different services, and get links to snapshots of that page at the time of access. This is mostly sufficient, so long as at least one service is trustworthy and doesn't shut down unexpectedly. One issue that might come up is that the archives might miss content if the user has to interact with the page a bunch in order to reveal it, but that's unlikely to be an issue for the sorts of things an academic would be ar
Read the rest
chiving. As another poster mentioned, some bibliography managers have this capability baked in, or can acquire it with a plugin. However, if you want more control, you might consider using something like `pywb`[1] to make your own web archive, and then host it yourself (e.g., on S3 or something). For more robustness, you could also publish those WARC files as torrents, or on IPFS, or if you're really cool, as part of the transaction messages on some blockchain (provided that you can stomach the transaction fees and don't anticipate having a large audience in China). The above will probably suffice so long as the Internet continues to function as a going concern. To accommodate concerns about what happens in the case of total societal collapse, I've been backing up the papers that I cite by converting to binary[2] and preserving the result via aluminum punchcards, which I then mail to Sam Altman since he's got a way better game plan for the end times than I do[3]. I'm not sure if aluminum is optimal for this purpose though, both for material science reasons and because I've been informed that I am under investigation for my role in some sort of aluminum smuggling ring. If that fails, my last backup plan entails constructing an allegorical representation of the papers in question (which is surprisingly easy-- the Hero's Journey from Joseph Campbell[4]'s exegesis is basically about differential equations anyways), which I will then craft into a compelling mythology and teach to the local youths. Hopefully this will ensure that the citations are preserved via oral tradition even in the wake of cataclysmic change. Figuring out the BibTeX for this archival format will be pretty tricky, though. But yeah, unless you have some extra aluminum lying around, `archivenow` or whatever's built-in to your reference manager should do the trick. --- 0. github.com/oduwsdl/archivenow 1. github.com/webrecorder/pywb 2. After stripping most of the markup removing images using `pandoc`, as aluminum is expensive. 3. https://www.bloomberg.com/features/2018-rich-new-zealand-doo... 4. https://en.wikipedia.org/wiki/Joseph_Campbell
In Open Questions about Generative Adversarial Networks
I arrived a little late here, but for anyone just getting into web archiving, it's a fascinating space! We maintain a master community index with a list of all the major archiving organizations and open source tools here: https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Comm... Some of the cooler projects apart from ArchiveBox.io: - https://github.com/webrecorder/pywb / webrecorder.io - https://github.com/HelloZeroNet/ZeroNet - https://github.com/oduwsdl/ipwb - https://getpolarized.io - https://github.com/internetarchive/brozzler And many more blog posts, articles, projects, and organizations on the wiki!
In Why there’s so little left of the early internet
Either curl or wget will get you pretty far. Learn one of them well. They are basically equivalent. I use curl. For current web apps, there is an interactive archiver written in Python, Web Recorder. It captures the full bi-directional traffic of a session. https://webrecorder.io/ Web Recorder uses an internal Python library, pywb. That might be a good place to look. https://github.com/webrecorder/pywb It looks like Selenium has done a lot of catching up on it's interface. I'd be curious how they compare now. Talk to librarians about archiving the web. They made Internet Archive and have a lot of experience.
In Ask HN: What is the best way to archive a webpage
Thanks Jack for mentioning Webrecorder! This is a project I started and it is now part of rhizome.org, a non-profit dedicated to promoting internet-based art and digital culture. I thought I’d add a few notes here, as there’s a few ways you can use Webrecorder and related tools. First, Webrecorder supports two distinct modes: - Native recording mode — http/s traffic goes to through the browser and is rewritten to point to the Webrecorder server (This is the default). - Remote browser mode — Webrecorder launches a browser in Docker container, and streams the screen to your browser (using noVNC). The traffic is either recorded or replayed depending on the mode, but the operation is the same (we call this ‘symmetrical archiving’) This gives you a recording proxy w/o having to configure your browser or install any plugins. You can choose this
Read the rest
mode by clicking the dropdown to choose a browser (currently Chrome and FF) This is essentially a remote browser configured via HTTP/S proxy, and allows us to record things like Flash and even Java applets, and other technologies that may become obsolete. - We also have a desktop player app, Webrecorder Player, available for download from: https://github.com/webrecorder/webrecorderplayer-electron/re... This is an app that plays back WARCs files (created by Webrecorder and elsewhere), and allows browsing any WARC file offline. Another way to create a web archive (for developers): You can use the devtools in the browser to export HAR files, and Webrecorder and Webrecorder Player will convert them on the fly and play them back. Unfortunately, this option is sort of limited for developers, but you can actually create a fairly good archive locally using HAR export (available in Chrome and Firefox at least). The conversion is done using this tool: https://github.com/webrecorder/har2warc - If you use webrecorder.io, you can register for an account or use it anonymously. If you register for an account, we provide 5GB storage and you have a permanent url for your archive. You can also upload existing WARCs (or HARs) - You can also run Webrecorder on your own! The main codebase is at: https://github.com/webrecorder/webrecorder and the remote browser system is actually a separate component and was first used for oldweb.today and lives at https://github.com/oldweb-today Finally, the core replay/recording tech is actually a separate component, an advanced ‘wayback machine’ being developed in https://github.com/ikreymer/pywb There’s a lot of different components here, and we would definitely appreciate help to any and all parts of the stack if anyone is interested! All our work is open-source and we are a non-profit, so any help is appreciated.
In Show HN: Tesoro – Personal internet archive

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.