RedPajama-Data

Yes > RedPajama-V2 is an open dataset for training large language models. The dataset includes over 100B text documents coming from 84 CommonCrawl snapshots and processed using the CCNet pipeline. https://github.com/togethercomputer/RedPajama-Data > Dolma Dataset: an open dataset of 3 trillion tokens from a diverse mix of web content, academic publications, code, books, and encyclopedic materials. https://allenai.github.io/dolma/

Gpt-4 was allegedly trained on 6T tokens for 2 epochs (i.e twice). 6T is far from exhausting the text data we have. Here's a 5 language 30T token dataset from web scrapped data. https://github.com/togethercomputer/RedPajama-Data And that is just web scrapped data. There are trillions of valuable tokens worth of text from the likes of pdfs/ebooks, academic journals and other documents that essentially has no web presence otherwise. https://annas-archive.org/llm >To improve performance using the same basic architecture the model need to scale both compute and training data. What you are trying to say here is that you need to scale both parameters and data as scaling data increases compute. That said, it's not really true. It would be best if you scaled both data and parameters but you can get increased performance just scaling one or the ot