MDmdaniel·Mar 7, 2024
relevant: https://github.com/ArchiveBox/ArchiveBox#readme and https://github.com/Rhizome-Conifer/conifer#readme (nee "webrecorder/webrecorder")
In YaCy, a distributed Web Search Engine, based on a peer-to-peer networkRhizome-Conifer
conifer
Python2016–2024
Rhizome-Conifer / conifer
conifer
Collect and revisit web pages.
11Distinct people
12Mentions
1Explicit recommenders
1.5KGitHub stars
What to know before exploring
What it is
Collect and revisit web pages.
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
For personal web archiving, I highly recommend http://webrecorder.io. The site lets you download archives in standard WARC format and play them back in an offline (Electron)…
JackC used a recommendation phrase. One indexed comment matches. This is a phrase match, not a verdict.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
- webrecorder/pywb · linked in 2 of the same discussions
Mentioned in the same Hacker News threads. That is not proof they are substitutes.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year12 dated mentions
Observed mentions: 2016-06-23 15:33:24 UTC to 2024-03-07 20:15:36 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2016 | 2016-01-01 00:00:00 UTC | 2017-01-01 00:00:00 UTC | 1 | 1 | 0 |
| 2017 | 2017-01-01 00:00:00 UTC | 2018-01-01 00:00:00 UTC | 3 | 0 | 3 |
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 4 | 0 | 4 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
12 sourcesI would personally recommend
https://github.com/rhizome-conifer/conifer
The intent is a webrecorder for the internet.
It records all js libraries, loads videos, and all else.
Once stored, you can review a snapshot at that point in time.
They have a service option, webrecorder.io, but this one let's you store directly locally.
In Ask HN: What browser extensions are a must-have in 2021?Now called https://github.com/Rhizome-Conifer/conifer ^
In Show HN: Self-hosted offline Internet from your browsing historyThis looks pretty cool, I'm excited to try it. The choice of lesser known format vs WARC is interesting, afaik WARC can do pretty complex stuff (like replaying youtube videos).
See: https://github.com/Rhizome-Conifer/conifer
In Show HN: Self-hosted offline Internet from your browsing historyA bit of scripting and this: https://archivebox.io/ might fit the bill?
Or this: https://github.com/Rhizome-Conifer/conifer
Maybe this: https://github.com/webrecorder/pywb
Or WWWOFFLE: http://www.gedanken.org.uk/software/wwwoffle/
Also pinboard.in but that's $
In Show HN: A bookmarking tool designed to help synthesize your web researchWhat's up with their Github? The link in the footer is broken: https://github.com/rhizome-conifer/conifer
And their Github organization has no public repositories. Is the code in the process of being moved from somewhere else?
In Conifer – collect and revisit web pagesThere's no reason why a web browser bookmark action doesn't automatically create a WARC (web archive) format.
Heck, with the cost of storage so low, recording every webpage you ever visit in searchable format is also very realistic. Imagine having the last 30 years of web browsing history saved on your local machine. This would especially be useful when in research mode and deep diving a topic.
[1] https://github.com/machawk1/warcreate
[2] https://github.com/machawk1/wail
[3] https://github.com/internetarchive/warcprox
EDIT: I forgot to mention https://github.com/webrecorder/webrecorder (the best general purpose web recorder application I have used during my previous research into archiving personal web usage)
In This Page is Designed to LastWhile what you say is true, the above method is the only method to archive arbitrary web pages. Yes it depends on user interactions to some extent, but it's possible to reasonably let a page load until fetches stop, and consider it rendered. Generally speaking, you can only archive some preset interactions with a modern web page. You can't hope to capture it all.
There are tools like WebRecorder[0] that do this to some extent by recording and replaying all requests. It's certainly a step in the right direction and demonstrates that the approach is viable. This was the only approach I tried that worked for archiving stuff like three.js demos. Worth mentioning there's also an Awesome list[1] that covers various web archival technologies.
[0] https://github.com/webrecorder/webrecorder
[1] https://github.com/iipc/awesome-web-archiving
In Archiving web sitesThanks Jack for mentioning Webrecorder! This is a project I started and it is now part of rhizome.org, a non-profit dedicated to promoting internet-based art and digital culture.
I thought I’d add a few notes here, as there’s a few ways you can use Webrecorder and related tools.
First, Webrecorder supports two distinct modes:
- Native recording mode — http/s traffic goes to through the browser and is rewritten to point to the Webrecorder server (This is the default).
- Remote browser mode — Webrecorder launches a browser in Docker container, and streams the screen to your browser (using noVNC). The traffic is either recorded or replayed depending on the mode, but the operation is the same (we call this ‘symmetrical archiving’) This gives you a recording proxy w/o having to configure your browser or install any plugins.
You can choose this
Read the rest
mode by clicking the dropdown to choose a browser (currently Chrome and FF)
This is essentially a remote browser configured via HTTP/S proxy, and allows us to record things like Flash and even Java applets, and other technologies that may become obsolete.
- We also have a desktop player app, Webrecorder Player, available for download from:
https://github.com/webrecorder/webrecorderplayer-electron/re...
This is an app that plays back WARCs files (created by Webrecorder and elsewhere), and allows browsing any WARC file offline.
Another way to create a web archive (for developers): You can use the devtools in the browser to export HAR files, and Webrecorder and Webrecorder Player will convert them on the fly and play them back.
Unfortunately, this option is sort of limited for developers, but you can actually create a fairly good archive locally using HAR export (available in Chrome and Firefox at least). The conversion is done using this tool: https://github.com/webrecorder/har2warc
- If you use webrecorder.io, you can register for an account or use it anonymously. If you register for an account, we provide 5GB storage and you have a permanent url for your archive. You can also upload existing WARCs (or HARs)
- You can also run Webrecorder on your own! The main codebase is at: https://github.com/webrecorder/webrecorder and the remote browser system is actually a separate component and was first used for oldweb.today and lives at https://github.com/oldweb-today
Finally, the core replay/recording tech is actually a separate component, an advanced ‘wayback machine’ being developed in https://github.com/ikreymer/pywb
There’s a lot of different components here, and we would definitely appreciate help to any and all parts of the stack if anyone is interested! All our work is open-source and we are a non-profit, so any help is appreciated.
https://github.com/webrecorder/webrecorder can be run using Docker. There's also plenty of Proxys that can save your browsing. See: http://netpreserve.org/projects/live-archiving-http-proxy/
In Show HN: Tesoro – Personal internet archiveFor personal web archiving, I highly recommend http://webrecorder.io. The site lets you download archives in standard WARC format and play them back in an offline (Electron) player. It's also open source and has a quick local setup via Docker - https://github.com/webrecorder/webrecorder .
Webrecorder is by a former Internet Archive engineer, Ilya Kreymer, who now captures online performance art for an art museum. What he's doing with capture and playback of Javascript, web video, streaming content, etc. is state of the art as far as I know.
(Disclaimer - I use bits of Webrecorder for my own archive, perma.cc.)
For OP, I would say consider building on and contributing back to Webrecorder -- or alternatively figure out what Webrecorder is good at and make sure you're good at something different. It's a crazy hard problem to do well and it's
Read the rest
great to have more ideas in the mix.
Webrecorder: Web Archiving for All
2 points · Open discussion