HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
internetarchive
sandcrawler
HTML2020–2021
internetarchive / sandcrawler

sandcrawler

Backend, IA-specific tools for crawling and processing the scholarly web. Content ends up in https://fatcat.wiki

Web DevelopmentAutomation & Workflows
Explore on GitHubRead Hacker News discussions
2Distinct people
2Mentions
0Explicit recommenders
28GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Backend, IA-specific tools for crawling and processing the scholarly web. Content ends up in https://fatcat.wiki

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by week2 dated mentions

Observed mentions: 2020-10-02 00:37:14 UTC to 2021-03-02 07:34:59 UTC. Weeks start Monday, UTC.

012020-09-28 00:00:00 UTC to 2020-10-05 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsSep 282020-10-05 00:00:00 UTC to 2020-10-12 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-10-12 00:00:00 UTC to 2020-10-19 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-10-19 00:00:00 UTC to 2020-10-26 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-10-26 00:00:00 UTC to 2020-11-02 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-11-02 00:00:00 UTC to 2020-11-09 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-11-09 00:00:00 UTC to 2020-11-16 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-11-16 00:00:00 UTC to 2020-11-23 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-11-23 00:00:00 UTC to 2020-11-30 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-11-30 00:00:00 UTC to 2020-12-07 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-12-07 00:00:00 UTC to 2020-12-14 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-12-14 00:00:00 UTC to 2020-12-21 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsDec 142020-12-21 00:00:00 UTC to 2020-12-28 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-12-28 00:00:00 UTC to 2021-01-04 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-01-04 00:00:00 UTC to 2021-01-11 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-01-11 00:00:00 UTC to 2021-01-18 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-01-18 00:00:00 UTC to 2021-01-25 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-01-25 00:00:00 UTC to 2021-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-02-01 00:00:00 UTC to 2021-02-08 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-02-08 00:00:00 UTC to 2021-02-15 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-02-15 00:00:00 UTC to 2021-02-22 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-02-22 00:00:00 UTC to 2021-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-03-01 00:00:00 UTC to 2021-03-08 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsMar 1

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Sep 282020-09-28 00:00:00 UTC2020-10-05 00:00:00 UTC110
Oct 52020-10-05 00:00:00 UTC2020-10-12 00:00:00 UTC000
Oct 122020-10-12 00:00:00 UTC2020-10-19 00:00:00 UTC000
Oct 192020-10-19 00:00:00 UTC2020-10-26 00:00:00 UTC000
Oct 262020-10-26 00:00:00 UTC2020-11-02 00:00:00 UTC000
Nov 22020-11-02 00:00:00 UTC2020-11-09 00:00:00 UTC000
Nov 92020-11-09 00:00:00 UTC2020-11-16 00:00:00 UTC000
Nov 162020-11-16 00:00:00 UTC2020-11-23 00:00:00 UTC000
Nov 232020-11-23 00:00:00 UTC2020-11-30 00:00:00 UTC000
Nov 302020-11-30 00:00:00 UTC2020-12-07 00:00:00 UTC000
Dec 72020-12-07 00:00:00 UTC2020-12-14 00:00:00 UTC000
Dec 142020-12-14 00:00:00 UTC2020-12-21 00:00:00 UTC000
Dec 212020-12-21 00:00:00 UTC2020-12-28 00:00:00 UTC000
Dec 282020-12-28 00:00:00 UTC2021-01-04 00:00:00 UTC000
Jan 42021-01-04 00:00:00 UTC2021-01-11 00:00:00 UTC000
Jan 112021-01-11 00:00:00 UTC2021-01-18 00:00:00 UTC000
Jan 182021-01-18 00:00:00 UTC2021-01-25 00:00:00 UTC000
Jan 252021-01-25 00:00:00 UTC2021-02-01 00:00:00 UTC000
Feb 12021-02-01 00:00:00 UTC2021-02-08 00:00:00 UTC000
Feb 82021-02-08 00:00:00 UTC2021-02-15 00:00:00 UTC000
Feb 152021-02-15 00:00:00 UTC2021-02-22 00:00:00 UTC000
Feb 222021-02-22 00:00:00 UTC2021-03-01 00:00:00 UTC000
Mar 12021-03-01 00:00:00 UTC2021-03-08 00:00:00 UTC101

What Hacker News says

2 sources
ddorian43·Mar 2, 2021
Check this experiment they did https://github.com/internetarchive/sandcrawler/blob/master/p... . This was just a part of the infrastructure. Point is ceph & friends have a lot of overhead. Example in ceph: by default, a file in S3 layer is split into 4MB chunks, and each of those chunks is replicated or erasure-coded. Using the same erasure coding as wasabi,b2-cloud, which is 16+4=20 (or 17+3=20), each of those 4MB chunks is split into 20 shards of ~200KB each. Each of those shards ends up having ~512B to 4KB of metadata. So from 10KB to 80KB of metadata for single 4MB chunk.
In Internet Archive Infrastructure

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.