TinyLlama

> they totally would have had it been technically feasible TinyLlama[1] has been made by an individual on their own last year, training a 1.1B model on 3T tokens with just 16 A100-40G GPUs in 90 days. It was definitely within reach of any funded org in 2019. In 2022 (IIRC), Google released the Chinchilla paper about the compute-optimal amount of data to train a given model, for a 1B model, the value was determined to be 20B tokens, which again is 3 orders of magnitude below the current state of the art for the same class of model. Until very recently (the first llama paper IIRC, and people noticing that the 7B model showed no sign of saturation during its already very long training) the ML community vastly underestimated the amount of training data that was needed to make a LLM perform at its potential. [1]: https://github.com/jzhang38/Ti

That claim is wrong. Training even a very basic model like TinyLlama takes 90 days on 16 x A100 GPUs [0]. A p4 box containing 8 X A100 GPUs costs $32/hr [1]. So 90 days * 2 p4 boxes * 24 hours * $32/hr = $138,240 Larger models run into the millions of dollars. Though I suppose it all depends on your personal definition of expensive. [0] https://github.com/jzhang38/TinyLlama [1] https://aws.amazon.com/ec2/instance-types/p4/

GitHub repo with links to the checkpoints: https://github.com/jzhang38/TinyLlama