HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
jcpeterson
openwebtext
Python2019–2019
jcpeterson / openwebtext

openwebtext

Open clone of OpenAI's unreleased WebText dataset scraper. This version uses pushshift.io files instead of the API for speed.

4Distinct people
4Mentions
0Explicit recommenders
769GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Open clone of OpenAI's unreleased WebText dataset scraper. This version uses pushshift.io files instead of the API for speed.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 10:02:05 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by 6-hour interval4 dated mentions

Observed mentions: 2019-02-26 20:29:04 UTC to 2019-03-02 18:21:09 UTC.

012019-02-26 18:00:00 UTC to 2019-02-27 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsFeb 26, 18:002019-02-27 00:00:00 UTC to 2019-02-27 06:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-02-27 06:00:00 UTC to 2019-02-27 12:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-02-27 12:00:00 UTC to 2019-02-27 18:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-02-27 18:00:00 UTC to 2019-02-28 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-02-28 00:00:00 UTC to 2019-02-28 06:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2019-02-28 06:00:00 UTC to 2019-02-28 12:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-02-28 12:00:00 UTC to 2019-02-28 18:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2019-02-28 18:00:00 UTC to 2019-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsFeb 28, 18:002019-03-01 00:00:00 UTC to 2019-03-01 06:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-03-01 06:00:00 UTC to 2019-03-01 12:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-03-01 12:00:00 UTC to 2019-03-01 18:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-03-01 18:00:00 UTC to 2019-03-02 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-03-02 00:00:00 UTC to 2019-03-02 06:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-03-02 06:00:00 UTC to 2019-03-02 12:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-03-02 12:00:00 UTC to 2019-03-02 18:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-03-02 18:00:00 UTC to 2019-03-03 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsMar 2, 18:00

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Feb 26, 18:002019-02-26 18:00:00 UTC2019-02-27 00:00:00 UTC101
Feb 27, 00:002019-02-27 00:00:00 UTC2019-02-27 06:00:00 UTC000
Feb 27, 06:002019-02-27 06:00:00 UTC2019-02-27 12:00:00 UTC000
Feb 27, 12:002019-02-27 12:00:00 UTC2019-02-27 18:00:00 UTC000
Feb 27, 18:002019-02-27 18:00:00 UTC2019-02-28 00:00:00 UTC000
Feb 28, 00:002019-02-28 00:00:00 UTC2019-02-28 06:00:00 UTC110
Feb 28, 06:002019-02-28 06:00:00 UTC2019-02-28 12:00:00 UTC000
Feb 28, 12:002019-02-28 12:00:00 UTC2019-02-28 18:00:00 UTC110
Feb 28, 18:002019-02-28 18:00:00 UTC2019-03-01 00:00:00 UTC000
Mar 1, 00:002019-03-01 00:00:00 UTC2019-03-01 06:00:00 UTC000
Mar 1, 06:002019-03-01 06:00:00 UTC2019-03-01 12:00:00 UTC000
Mar 1, 12:002019-03-01 12:00:00 UTC2019-03-01 18:00:00 UTC000
Mar 1, 18:002019-03-01 18:00:00 UTC2019-03-02 00:00:00 UTC000
Mar 2, 00:002019-03-02 00:00:00 UTC2019-03-02 06:00:00 UTC000
Mar 2, 06:002019-03-02 06:00:00 UTC2019-03-02 12:00:00 UTC000
Mar 2, 12:002019-03-02 12:00:00 UTC2019-03-02 18:00:00 UTC000
Mar 2, 18:002019-03-02 18:00:00 UTC2019-03-03 00:00:00 UTC110

What Hacker News says

1 sources
I have been working with a group that is trying to clone this dataset and make it publicly available (https://github.com/jcpeterson/openwebtext), and I have noticed quite a bit of code in the scraped dataset. Future releases of our dataset will be pre-filtered with another LSTM language model that will filter sentences by their probability under more conversational / literary datasets.
In PHP code generated by GPT-2

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.