NInickvincent·Mar 1, 2021
I'm curious about this as well!
Search results sent me to this Twitter thread which suggests it was still a mystery in Aug 2020:
https://twitter.com/vboykis/status/1290030614410702848?lang=...
To speculate a bit:
The GPT-3 paper (section 2.2) mentions using two datasets referred to as "books1" and "books2", which are 12B and 55B byte pair encoded tokens each.
Project Gutenberg has 3B word tokens I believe, so it seems like it could be one of them, assuming the ratio of word tokens to byte-pair tokens is something like 3:12 to 3:55.
Another likely candidate alongside Gutenberg is libgen, apparently, and looks like there have been successful efforts to create a similar dataset called bookcorpus: https://github.com/soskek/bookcorpus/issues/27). The discussion on that github issue suggests bookcorpus is very similar to "books2", which would m
Read the rest
ake gutenberg "books1"?
This might be why the paper is intentionally vague about the books used?