HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
ArchiveTeam
grab-site
Python2016–2026
ArchiveTeam / grab-site

grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

9Distinct people
22Mentions
0Explicit recommenders
1.6KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 7:24:48 PM UTC.

The Apple Support Community will permanently close on November 20, 2026 direct repository linkCritical

The Apple Support Community is being shut down due to its long-standing reputation for providing inaccurate, unhelpful, and toxic responses rather than genuine technical support. Users criticize the platform for polluting search results with misinformation and serving as a venue for community members to harass or dismiss users instead of solving problems.

234 readable replies · Supporting reply 1 · Supporting reply 2 · Supporting reply 3 · Supporting reply 4

Mentions over time

HN mentions by year22 dated mentions

Observed mentions: 2016-02-08 10:51:40 UTC to 2026-07-06 18:23:08 UTC.

02462016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments20162017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 4 mentions; 0 submissions, 4 comments2020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments20212022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 6 mentions; 0 submissions, 6 comments2025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2026-01-01 00:00:00 UTC to 2027-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments. Incomplete calendar period.2026*

* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC202
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC000
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC202
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC404
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC202
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC000
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC202
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC101
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC606
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC101
2026 (partial)2026-01-01 00:00:00 UTC2027-01-01 00:00:00 UTC202

What Hacker News says

2 sources
author here. AMA. since I wrote this, i started experimenting with grab-site: https://github.com/ludios/grab-site it's a wrapper around (and a fork of) wpull but the main advantage over it is it can do on-the-fly reconfiguration of delay, concurrency, ignore patterns and so on. it also provides a nice web interface. if you're only crawling one site every once in a while, wpull and crawl are fine, but for larger projects, grab-site is a must.
In Archiving web sites

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.