HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
minimaxir
get-all-hacker-news-submissions-comments
Python2015–2018
minimaxir / get-all-hacker-news-submissions-comments

get-all-hacker-news-submissions-comments

Simple Python scripts to download all Hacker News submissions and comments and store them in a PostgreSQL database.

2Distinct people
13Mentions
0Explicit recommenders
127GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Simple Python scripts to download all Hacker News submissions and comments and store them in a PostgreSQL database.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 4:43:50 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year13 dated mentions

Observed mentions: 2015-05-11 14:59:02 UTC to 2018-02-19 05:26:05 UTC.

024682015-01-01 00:00:00 UTC to 2016-01-01 00:00:00 UTC (exclusive): 8 mentions; 2 submissions, 6 comments20152016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments20162017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2018

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20152015-01-01 00:00:00 UTC2016-01-01 00:00:00 UTC826
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC202
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC202
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC101

What Hacker News says

8 sources
minimaxir·Oct 23, 2015Submission

Tell HN: Full Hacker News dataset now available on BigQuery

https://bigquery.cloud.google.com/table/fh-bigquery:hackernews.comments The dataset is up-to-date for October 2015 and uses the official HN API as a data source. (Unfortuntely, this includes the HTML formatting in the comments) Dataset is about 4GB total; since BigQuery allows for 1000GB processing for free each month, it is effectively no cost to analyze. Felipe Hoffa (who uploaded the dataset) has links to the dataset along with some sample Python code for analysis: https://github.com/fhoffa/notebooks/blob/master/analyzing%20hacker%20news.ipynb I have a few scripts for downloading all the data manually (https://github.com/minimaxir/get-all-hacker-news-submissions-comments ), but the BigQuery table may be more pragmatic. I can do some more sample queries if requested. (See my BigQuery tutorial for Reddit data: http://minimaxir.com/2015/1
Read the rest
0/reddit-bigquery/ )
238 points · Open discussion
Relevant: if you're looking for a way to scrape all Hacker News submissions and comments, I have a GitHub repository containing Python scripts showing how to do just that: https://github.com/minimaxir/get-all-hacker-news-submissions... With respect to the article, I've done data analysis on Hacker News myself (e.g. http://minimaxir.com/2014/10/hn-comments-about-comments/ ) and I'm a little skeptical. I was not a fan of using NTLK to classify positive/negative because the models used for training sentiment are based on reviews, not internet site comments, and when I had used NTLK, the results were skewed significantly. I used an alternate model mentioned in the linked post which IMO worked better. The examples used in the article seem more like correlation-implying-causation and cherry-picking. There are many, many counterexamples of start
Read the rest
ups people loved on HN which died a painful death.
In Hacker News as a case study to test the wisdom of the crowd theory
> the difficulty is that investigating a complex dataset is by definition too complex to really fit into a blog post It's not complex, but it requires answering scaleability questions. How do you fit the data into memory? How do you even get big data? I managed to analyze 5.6M Hacker News comments (http://minimaxir.com/2014/10/hn-comments-about-comments/ ) by writing a scraper which dumps everything into a PostgreSQL database (https://github.com/minimaxir/get-all-hacker-news-submissions... ) because it's way too big to fit in memory. Likewise, to visualize the recent NYC taxi data, (http://minimaxir.com/2015/08/nyc-map/ ), I just used BigQuery and saved myself a load of time downloading the data and setting things up. But in many cases, data is not easily accessible, and not cheap to store and process. I want to switch to an actual big da
Read the rest
ta workflow with Spark, but running the numbers with EC2 storage and processing, it won't be cheap and I may have to set up a Patreon to subsidize things.
In Statistics for Hackers

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.