HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
minimaxir
get-all-hacker-news-submissions-comments
Python2015–2018
minimaxir / get-all-hacker-news-submissions-comments

get-all-hacker-news-submissions-comments

Simple Python scripts to download all Hacker News submissions and comments and store them in a PostgreSQL database.

2Distinct people
13Mentions
0Explicit recommenders
127GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Simple Python scripts to download all Hacker News submissions and comments and store them in a PostgreSQL database.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 10:49:33 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year13 dated mentions

Observed mentions: 2015-05-11 14:59:02 UTC to 2018-02-19 05:26:05 UTC.

024682015-01-01 00:00:00 UTC to 2016-01-01 00:00:00 UTC (exclusive): 8 mentions; 2 submissions, 6 comments20152016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments20162017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2018

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20152015-01-01 00:00:00 UTC2016-01-01 00:00:00 UTC826
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC202
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC202
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC101

What Hacker News says

2 sources
A couple years ago, I did my own analysis of all Hacker News submissions (http://minimaxir.com/2014/02/hacking-hacker-news/) and also wrote a script around that time to get all data (https://github.com/minimaxir/get-all-hacker-news-submissions... , see also a modern dataset on Kaggle derived from it: https://www.kaggle.com/hacker-news/hacker-news-posts). I only looked at the # of points as a metric for quality, so front v. back with this approach is interesting. Given the good work in this post, I may take another look at the data myself. This is a case where the sample size used may be problematic. "425 fronts against 570 corresponding backs" (n = 995), in the grand scheme of Hacker News, is not a lot, even if statistical analysis permits it (example: the by-hour Chi-Sq test, which barely hits the 5-per-cell assumption). Given the method
Read the rest
of collection by scraping the front page directly, this is understandable, though. However, that presents a problem. the front-page algorithm has changed in recent months and I myself have had difficulty predicting what makes the front page and what doesn't (and what ends up making the front page hours after being submitted for no reason). With relatively new features like the second-chance pool and explicit dupe marking, there is new quality control of the front page thanks to dang/sctb. That is another issue of looking at a small subset of HN data; it does not reflect the site as a whole, although looking at more-recent data might be more beneficial for optimizing one's own posts.
In Data Mining Hacker News: Front vs. Back
> At 4 GB, I'd just as soon query this locally, but this looks like a fun exercise. This requires scraping all the Hacker News data manually, for which I have a tool to do so (https://github.com/minimaxir/get-all-hacker-news-submissions...) which I mentioned in the post you linked, but it still requires a significant amount of time to get/process the data, hence why the BigQuery dataset has a significant advantage.
In Top Books on Amazon Based on Links in Hacker News Comments

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.