internetarchive

brozzler

Python2018–2019

internetarchive / brozzler

brozzler

brozzler - distributed browser-based web crawler

Explore on GitHub Read Hacker News discussions

4Distinct people

4Mentions

0Explicit recommenders

823GitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

brozzler - distributed browser-based web crawler

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month4 dated mentions

Observed mentions: 2018-06-05 22:41:18 UTC to 2019-10-19 08:52:52 UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Comments
Jun 2018	2018-06-01 00:00:00 UTC	2018-07-01 00:00:00 UTC	1	1
Jul 2018	2018-07-01 00:00:00 UTC	2018-08-01 00:00:00 UTC	0	0
Aug 2018	2018-08-01 00:00:00 UTC	2018-09-01 00:00:00 UTC	0	0
Sep 2018	2018-09-01 00:00:00 UTC	2018-10-01 00:00:00 UTC	0	0
Oct 2018	2018-10-01 00:00:00 UTC	2018-11-01 00:00:00 UTC	0	0
Nov 2018	2018-11-01 00:00:00 UTC	2018-12-01 00:00:00 UTC	0	0
Dec 2018	2018-12-01 00:00:00 UTC	2019-01-01 00:00:00 UTC	0	0
Jan 2019	2019-01-01 00:00:00 UTC	2019-02-01 00:00:00 UTC	0	0
Feb 2019	2019-02-01 00:00:00 UTC	2019-03-01 00:00:00 UTC	0	0
Mar 2019	2019-03-01 00:00:00 UTC	2019-04-01 00:00:00 UTC	0	0
Apr 2019	2019-04-01 00:00:00 UTC	2019-05-01 00:00:00 UTC	1	1
May 2019	2019-05-01 00:00:00 UTC	2019-06-01 00:00:00 UTC	0	0
Jun 2019	2019-06-01 00:00:00 UTC	2019-07-01 00:00:00 UTC	0	0
Jul 2019	2019-07-01 00:00:00 UTC	2019-08-01 00:00:00 UTC	0	0
Aug 2019	2019-08-01 00:00:00 UTC	2019-09-01 00:00:00 UTC	0	0
Sep 2019	2019-09-01 00:00:00 UTC	2019-10-01 00:00:00 UTC	0	0
Oct 2019	2019-10-01 00:00:00 UTC	2019-11-01 00:00:00 UTC	2	2

What Hacker News says

4 sources

dredmorbius·Oct 19, 2019

Disclaimer: I'm not directly involved in any of this, though I've followed some general discussions. 1. The Web is much less diverse than you think it is, for two reasons. 2. Zipf/Power functions: the largest sites are a huge part of the Web. If people are spending (say) an hour online per day, and 40 minutes of that are FB/Insta, then all the rest of the Web gets 20 minutes. And much of that will be the next most popular site. Attention is fundamentally limited. In terms of experience, the Web is fairly small. 3. Which of course isn't content, but here again, much of the original content only hits a relatively small number of sites. I have looked into news site stats, and what you'll see for national papers of record (NYT, WSJ, WashPo, etc.) is about 150-500 items per day (Sundays are highest). The news services (AP, UPI, AFP, Re

Read the rest

uters) tend to run 1,000 - 5,000 items/day. User-generated content sites are obviously higher than this, though much of that content is derivative (sourced/linked elsewhere). 4. Web engines. Even with a diversity of sites, there are a limited number of systems driving this. I've been noting a consolidation around news sites of a number of standard templates, and of course FB, YT, Instagram, Reddit, etc., will get you a bunch of specific Web engines. Which means it's less "one site per archiver" and more "what sites does this archiver work on?". And worse: over what period of time -- just because an archiver works on www.example.com from 2018-1-1 to 2018-12-31 doesn't mean it works from 2019-1-1 onward. There are a few other approaches. Archive.today (archive.is / archive.fo) basically translate pages to their own representation. There are standardised presentation engines based on Readability (now defunct as a site, though its parser remains), used by Mozilla's Pocket and other tools (InstaPaper, etc.). 5. I also suspect we may see at least some normalisation around HTML5 (or subsequent) elements. Which I'd really like to have happen. NY Times, for example, has abandoned <table> for its own custom, and horribly broken, table-presentation alternative crap. 6. Yes, this leaves one-off SPAs which may be hard to archive. I don't know what the plans are for that. (Now to see what IA have had to say about any of this.) Update: from an IA response in this thread, "brozzler": https://github.com/internetarchive/brozzler

In The Wayback Machine: Fighting Digital Extinction in New Ways

dvanduzer·Oct 18, 2019

A crawler has two high level options: parse the page, or render the page. Most of our parser-based crawling is done by Heritrix (crawler.archive.org) and most of our render-based crawling is done by a proxy-based recorder similar to what you theorize (https://github.com/internetarchive/brozzler).

In The Wayback Machine: Fighting Digital Extinction in New Ways

nikisweeting·Apr 4, 2019

I arrived a little late here, but for anyone just getting into web archiving, it's a fascinating space! We maintain a master community index with a list of all the major archiving organizations and open source tools here: https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Comm... Some of the cooler projects apart from ArchiveBox.io: - https://github.com/webrecorder/pywb / webrecorder.io - https://github.com/HelloZeroNet/ZeroNet - https://github.com/oduwsdl/ipwb - https://getpolarized.io - https://github.com/internetarchive/brozzler And many more blog posts, articles, projects, and organizations on the wiki!

In Why there’s so little left of the early internet

bnewbold·Jun 5, 2018

I can't speak formally for the Internet Archive, but the existing content and services are not going to disappear overnight: funding comes from several sources, thought has been put in to organizational structure, and things have been designed to keep core access and preservation infrastructure running with minimal cost and effort (eg, if the economy tanks). Getting the content coverage people sometimes assume we already have is another matter. Additional funding (thanks for you donation!) go towards additional crawling and keeping up with the endless treadmill of media types and protocols. Eg, headless browser crawling development and deployment to capture javascript-heavy sites (https://github.com/internetarchive/brozzler); this is much more expensive than "classic" crawling. For more on increasing storage costs and the under-funded sta

Read the rest

te of web archiving in general, I recommend David Rosenthal's blog, eg: https://blog.dshr.org/2018/05/longer-talk-at-msst2018.html https://blog.dshr.org/2014/03/the-half-empty-archive.html Far more effective and robust than hoping the archive is "suck it up for us" is to upload snapshots/dumps/exports yourself! Anybody can create an archive.org account and upload content (recommend https://github.com/jjjake/internetarchive over the HTML form), within reasonable limits. Obviously, care needs to be taken to remove sensitive (and personal) information first.

In Archive.org and California to start a data sharing and preservation project