soskek

bookcorpus

Python2020–2024

soskek / bookcorpus

bookcorpus

Crawl BookCorpus

AI & Machine Learning Automation & Workflows

Explore on GitHub Read Hacker News discussions

4Distinct people

4Mentions

0Explicit recommenders

866GitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

Crawl BookCorpus

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year4 dated mentions

Observed mentions: 2020-10-25 21:16:36 UTC to 2024-05-28 20:31:02 UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Submissions	Comments
2020	2020-01-01 00:00:00 UTC	2021-01-01 00:00:00 UTC	1	0	1
2021	2021-01-01 00:00:00 UTC	2022-01-01 00:00:00 UTC	1	0	1
2022	2022-01-01 00:00:00 UTC	2023-01-01 00:00:00 UTC	0	0	0
2023	2023-01-01 00:00:00 UTC	2024-01-01 00:00:00 UTC	1	1	0
2024	2024-01-01 00:00:00 UTC	2025-01-01 00:00:00 UTC	1	0	1

What Hacker News says

4 sources

Intralexical·May 28, 2024

It is apparently widely suspected that a certain "Books2" dataset mentioned by OpenAI is basically just LibGen: https://blusharkmedia.medium.com/the-ongoing-battle-against-... https://techhq.com/2023/09/can-libgen-shadow-library-survive... https://www.twitter.com/theshawwn/status/1320282152689336320 https://qz.com/openai-books-piracy-microsoft-meta-google-cha... https://qz.com/shadow-libraries-are-at-the-heart-of-the-moun... https://goodereader.com/blog/e-book-news/authors-file-lawsui... When asked about whether this was true, they refused to answer based on confidentiality concerns, then said they had deleted all copies of the dataset, stopped using it, and no longer employed the individuals that compiled it: https://www.businessinsider.com/openai-destroyed-ai-training... We do know for a fact that the (non-OpenAI-controlled) "Books3" da

Read the rest

taset is just "all of bibliotik": https://www.twitter.com/theshawwn/status/1320282149329784833 https://github.com/soskek/bookcorpus/issues/27 And we also apparently know for a fact that this was included in the datasets used to train LLAMA: https://en.wikipedia.org/wiki/The_Pile_(dataset) https://aicopyright.substack.com/p/the-books-used-to-train-l... https://aicopyright.substack.com/p/has-your-book-been-used-t...

In The Internet Archive is under a DDoS attack

superpirate·Sep 8, 2023Submission

Show HN: New AI Dataset Based on LibGen and Sci-Hub

We recently began extracting the text layers of scholarly publications and books to include in our database. This encompasses sources such as scimag, libgen, and the latest zlib leaks. Our project, named the Standard Template Construct, also features a distributed search engine and incorporates various AI routines to handle the text corpus. Today we have releases our first dataset, STC230908. This dataset contains approximately 75,000 book texts, 1.3 million scholarly paper texts, and 24 million abstracts, including the years from 2021 to 2023. We're currently in the process of preparing the next version of the dataset, which will include an additional 300,000 books. How to Access Short Instructions: Install IPFS and launch it. pip3 install stc-geck && geck - documents More details: the dataset is released in IPFS and replicated to multip

Read the rest

le nodes. It is in format of database for the search engine that we use in STC. GECK is the library that embeds this search engine and allows to stream all contained data in easy way. Even more detailed Instructions: https://github.com/nexus-stc/stc/tree/master/geck

4 points · Open discussion

nickvincent·Mar 1, 2021

I'm curious about this as well! Search results sent me to this Twitter thread which suggests it was still a mystery in Aug 2020: https://twitter.com/vboykis/status/1290030614410702848?lang=... To speculate a bit: The GPT-3 paper (section 2.2) mentions using two datasets referred to as "books1" and "books2", which are 12B and 55B byte pair encoded tokens each. Project Gutenberg has 3B word tokens I believe, so it seems like it could be one of them, assuming the ratio of word tokens to byte-pair tokens is something like 3:12 to 3:55. Another likely candidate alongside Gutenberg is libgen, apparently, and looks like there have been successful efforts to create a similar dataset called bookcorpus: https://github.com/soskek/bookcorpus/issues/27). The discussion on that github issue suggests bookcorpus is very similar to "books2", which would m

Read the rest

ake gutenberg "books1"? This might be why the paper is intentionally vague about the books used?

In On the Dangers of Stochastic Parrots [pdf]

sillysaurusx·Oct 25, 2020

Hello! I made the books3 dataset. I am delighted to hear that this might be useful for your personal use. I spent four or five days intensely working on the script to convert an .epub to .txt. It's a surprisingly hard problem, so I tried to be correct in every detail. It was important to me that an AI learn the way humans learn. Therefore, yes, I have found that these books are quite readable. However there are some unique challenges due to the nature of it being .txt, which I would love to help you navigate. Firstly, the top and the bottom of each file is usually "repetitive". By that I mean, there tends to be a lot of metadata that might be hard to skip. However, the neat thing is, these aren't actually .txt files -- it's Markdown. And we can take advantage of the markdown format to help you out. Here is a direct link to "A Mathematicia

Read the rest

n's Apology": https://gist.githubusercontent.com/shawwn/1b399325e866731165... I like this file as an example, because it illustrates both problems that will be challenging for you. Firstly, the book itself doesn't begin until halfway down the file. There is a ridiculously long "forward" section, written by someone else. Normally this would be easy to skip. The cheat code I use to read this book is to search for "# 1". Double quote, followed by pound sign (or "hashtag" as the kids say nowadays, ha), followed by a space, then the number one. That brings you straight to chapter one. However, not all books use that format for chapter one. So I would recommend searching for a regex, if you can: "beginning of line" followed by pound sign. That'll usually do the trick. And yes, I find all of these to be quite readable, which I was proud of. The code is courtesy of Aaron Swartz, by the way. I merely refined it. I am so proud of him for what he was able to achieve during his lifetime. This is "html-to-text" with some modifications. If you want to convert .epub files to .txt for your own use, the script is here: https://github.com/shawwn/scrap/blob/master/epub2txt-all It has a rather cryptic name of "epub2txt-all". Sorry about that. But the script itself should be just a matter of running it. Here are some notes on what I did with this script, in case you find it helpful: https://github.com/soskek/bookcorpus/issues/27 Happy reading! Please let me know any other questions you might have. By the way, dmca.mp4 is a joke video. Sort of. It's a good question of what is actually going on there. As far as I can tell, it's a dozen professionals who have gathered together to simulate jacking themselves off while singing "ahhhhhhHHhhH" for ten minutes. I don't really know anything beyond that, but the-eye.eu community seems rather proud of it.

In Books in .txt format for AI training purposes