Show HN: New AI Dataset Based on LibGen and Sci-Hub
soskek
bookcorpus
Python2020–2024
soskek / bookcorpus
bookcorpus
Crawl BookCorpus
4Distinct people
4Mentions
0Explicit recommenders
866GitHub stars
What to know before exploring
What it is
Crawl BookCorpus
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 10:50:02 AM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year4 dated mentions
Observed mentions: 2020-10-25 21:16:36 UTC to 2024-05-28 20:31:02 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 1 | 1 | 0 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
1 sourcesSUsuperpirate·Sep 8, 2023Submission
We recently began extracting the text layers of scholarly publications and books to include in our database. This encompasses sources such as scimag, libgen, and the latest zlib leaks. Our project, named the Standard Template Construct, also features a distributed search engine and incorporates various AI routines to handle the text corpus.
Today we have releases our first dataset, STC230908. This dataset contains approximately 75,000 book texts, 1.3 million scholarly paper texts, and 24 million abstracts, including the years from 2021 to 2023.
We're currently in the process of preparing the next version of the dataset, which will include an additional 300,000 books.
How to Access
Short Instructions:
Install IPFS and launch it.
pip3 install stc-geck && geck - documents
More details: the dataset is released in IPFS and replicated to multip
le nodes. It is in format of database for the search engine that we use in STC. GECK is the library that embeds this search engine and allows to stream all contained data in easy way.
Even more detailed Instructions: https://github.com/nexus-stc/stc/tree/master/geck