get-all-hacker-news-submissions-comments

You could do that with the Algolia API: https://github.com/minimaxir/get-all-hacker-news-submissions... But as noted, BigQuery is more pragmatic.

Not quite what you asked: https://github.com/minimaxir/get-all-hacker-news-submissions... source: https://news.ycombinator.com/item?id=14337275 search related: https://hn.algolia.com/?query=bigquery-public-data:hacker_ne... found: https://news.ycombinator.com/item?id=14580698 (New HN data dump available with over 14.5m entries) 7.3GB through 2017Jun) https://news.ycombinator.com/item?id=10440502 (Tell HN: Full Hacker News dataset now available on BigQuery)

Direct URL to project details: https://devpost.com/software/tagger-news A few comments: 1. To other commenters, as with the HN Vue demo a week ago (https://news.ycombinator.com/item?id=14284877), the project is a technical proof-of-concept; the aesthetics aren't the primary focus. 2. The Algolia API is better for scraping because it allows for bulk requests, unlike the official API (my old 2014 script still works I think: https://github.com/minimaxir/get-all-hacker-news-submissions...) 3) How much time did it take to manually label the training/test set before training the RF classifier? Even with topic modeling for extrapolating tags, accurate labeling for 20,000 submissions is a task.

A couple years ago, I did my own analysis of all Hacker News submissions (http://minimaxir.com/2014/02/hacking-hacker-news/) and also wrote a script around that time to get all data (https://github.com/minimaxir/get-all-hacker-news-submissions... , see also a modern dataset on Kaggle derived from it: https://www.kaggle.com/hacker-news/hacker-news-posts). I only looked at the # of points as a metric for quality, so front v. back with this approach is interesting. Given the good work in this post, I may take another look at the data myself. This is a case where the sample size used may be problematic. "425 fronts against 570 corresponding backs" (n = 995), in the grand scheme of Hacker News, is not a lot, even if statistical analysis permits it (example: the by-hour Chi-Sq test, which barely hits the 5-per-cell assumption). Given the method

> At 4 GB, I'd just as soon query this locally, but this looks like a fun exercise. This requires scraping all the Hacker News data manually, for which I have a tool to do so (https://github.com/minimaxir/get-all-hacker-news-submissions...) which I mentioned in the post you linked, but it still requires a significant amount of time to get/process the data, hence why the BigQuery dataset has a significant advantage.

Clickables: https://bigquery.cloud.google.com/table/fh-bigquery:hackerne... https://github.com/fhoffa/notebooks/blob/master/analyzing%20... https://github.com/minimaxir/get-all-hacker-news-submissions... http://minimaxir.com/2015/10/reddit-bigquery/

https://bigquery.cloud.google.com/table/fh-bigquery:hackernews.comments The dataset is up-to-date for October 2015 and uses the official HN API as a data source. (Unfortuntely, this includes the HTML formatting in the comments) Dataset is about 4GB total; since BigQuery allows for 1000GB processing for free each month, it is effectively no cost to analyze. Felipe Hoffa (who uploaded the dataset) has links to the dataset along with some sample Python code for analysis: https://github.com/fhoffa/notebooks/blob/master/analyzing%20hacker%20news.ipynb I have a few scripts for downloading all the data manually (https://github.com/minimaxir/get-all-hacker-news-submissions-comments ), but the BigQuery table may be more pragmatic. I can do some more sample queries if requested. (See my BigQuery tutorial for Reddit data: http://minimaxir.com/2015/1

Relevant: if you're looking for a way to scrape all Hacker News submissions and comments, I have a GitHub repository containing Python scripts showing how to do just that: https://github.com/minimaxir/get-all-hacker-news-submissions... With respect to the article, I've done data analysis on Hacker News myself (e.g. http://minimaxir.com/2014/10/hn-comments-about-comments/ ) and I'm a little skeptical. I was not a fan of using NTLK to classify positive/negative because the models used for training sentiment are based on reviews, not internet site comments, and when I had used NTLK, the results were skewed significantly. I used an alternate model mentioned in the linked post which IMO worked better. The examples used in the article seem more like correlation-implying-causation and cherry-picking. There are many, many counterexamples of start

> the difficulty is that investigating a complex dataset is by definition too complex to really fit into a blog post It's not complex, but it requires answering scaleability questions. How do you fit the data into memory? How do you even get big data? I managed to analyze 5.6M Hacker News comments (http://minimaxir.com/2014/10/hn-comments-about-comments/ ) by writing a scraper which dumps everything into a PostgreSQL database (https://github.com/minimaxir/get-all-hacker-news-submissions... ) because it's way too big to fit in memory. Likewise, to visualize the recent NYC taxi data, (http://minimaxir.com/2015/08/nyc-map/ ), I just used BigQuery and saved myself a load of time downloading the data and setting things up. But in many cases, data is not easily accessible, and not cheap to store and process. I want to switch to an actual big da

I have a repository of scripts for downloading all Hacker News data using Python and storing in PostgreSQL: https://github.com/minimaxir/get-all-hacker-news-submissions...

I have a GitHub repository for getting all the HN data the hard way using Python and the Algolia API, and storing them in PostgreSQL: https://github.com/minimaxir/get-all-hacker-news-submissions...

It's worth nothing that according to the source code, this uses the Algolia HN API and not the official HN API, which is a smart move because the official API still doesn't have bulk requests. As a slightly off-topic aside, I have a GitHub repo showing how to download all Hacker News stories and comments using the Algolia API: https://github.com/minimaxir/get-all-hacker-news-submissions...