HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
ipfs
distributed-wikipedia-mirror
TypeScript2017–2022
ipfs / distributed-wikipedia-mirror · Archived

distributed-wikipedia-mirror

[ ARCHIVED ] Wikipedia snapshots on IPFS (2017-2021). The ZIM-unpacking approach is a dead end, read the README for what to build instead.

9Distinct people
9Mentions
0Explicit recommenders
684GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

[ ARCHIVED ] Wikipedia snapshots on IPFS (2017-2021). The ZIM-unpacking approach is a dead end, read the README for what to build instead.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year9 dated mentions

Observed mentions: 2017-05-07 22:33:18 UTC to 2022-09-13 19:33:34 UTC.

012342017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments20172018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 4 mentions; 1 submissions, 3 comments20192020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2022

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC101
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC101
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC413
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC000
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC202
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC110

What Hacker News says

9 sources
While building the Wikipedia mirror on IPFS (with search), we tried using the dumps from Wikipedia themselves but ended up using Zim archives from kiwix.org instead. The end result is here: https://github.com/ipfs/distributed-wikipedia-mirror For actually ingesting the archives, dignifiedquire expanded a Rust utility aptly named Zim, which you can find here https://github.com/dignifiedquire/zim Both repos contain information (and code of course) on how to extract information from the Zim archives
In Ask HN: Has anyone had luck using the Wikipedia data dumps?
> WMF sites being a thing that have a shared telecommunications Single Point of Failure In fact, one of the long-term projects in WMF is making sure the infrastructure is resistant to single-point-of-failure problems - up to whole data center going down. We are pretty close to it (not sure if 100%, but if not close to it). Of course, if you consider existence of WMF to be point of failure, it's another question, by that logic existence of Wikipedia can be treated as single point of failure too. Anybody is welcome to create a new Wikipedia, but that's certainly not a point of criticism towards WMF. > It was able to be shut down because it had an SPOF: Napster-the-corporation owned and maintained all the "supernodes" WMF does not own the content or the code, both are in open access and extensively mirrored. WMF does own the hardware - I don
Read the rest
't think there's a way to do anything about it, unless somebody wants to donate a data center :) > If the Great Firewall can block your site/content entirely with a single rule, you have an SPOF. True, though there are ways around it. Currently witnesses with Turkey blocking Wikipedia. See e.g. https://github.com/ipfs/distributed-wikipedia-mirror for ways around it. > Instead, to answer such problems, you need true distribution. I am skeptical about the possibility of making community work using "true distribution". Even though we have good means to distribute hardware and software, be it production or development code, we still do not have any ways to make a community without having gathering points. I won't say it is impossible. I'd say I have yet to see anybody having done it. But if somebody wants to try, all power to them. You can read more about Wikimedia discussions on the topic here: https://strategy.wikimedia.org/wiki/Proposal:Distributed_Wik... > such that you can "bring Wikipedia with you" That already exists, there are several offline wikipedia setups and projects: https://strategy.wikimedia.org/wiki/Offline/List_of_Offline_... > this can take the form of preloaded "Wikipedia mirror in a box" appliances We are pretty close to this - you can install working Mediawiki setup very quickly (vagrant or I think there are some other containers too, I use vagrant), dumps are there. Won't be 100% copy of true site since there are some complex operational structures that ensure caching, high availability, etc. which kinda hard to put into a box - they are in public (mostly as puppet recipes) but implementing them is not out-of-the-box experience. But you can make a simple mirror with relatively low effort (probably several hours excluding time to actually load the data, that depends on how beefy your hardware is :) Most of this, btw, is made possible by the work of WMF Engineers :)
In Wikimedia Foundation spending

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.