YaLM-100B

How are we going to take back the LLM space from centralized nerfing by OpenAI? The unrestricted GPT-3 back in 2021 hooked up to the back of AI adventure was incredible. The Yandex YaLM LLM model is 100B parameters but is unfortunately 75% in Russian (See https://github.com/yandex/YaLM-100B). You can technically run it though on a big AWS instance. Someone needs to invent some good LLM compression to get the model to fit in under 16gb so we can run LLMs at home or someone needs to start releasing GPUs with huge amounts of RAM. I think 16GB of ram was not a big deal 10 years ago, how have we not progressed?

I don't much follow AI news beyond what I randomly happen to see on HN, but this might still be the largest open source model: https://github.com/yandex/YaLM-100B . There's discussion of it here: https://old.reddit.com/r/MachineLearning/comments/vpn0r1/d_h... - at the bottom of that page is a comment from someone who actually ran it in the cloud.

I really enjoyed this interview [1] with Emad Mostaque who I gather is probably the key funding and organization player in that stuff. It remains to be seen exactly how "open" it winds up playing out over time, but they're talking a compelling game and the EleutherAI people seem to be pretty heavily involved, which you probably wouldn't do if you weren't serious. Yandex has also put up a ~100B language model [2]. My old colleagues at Meta have also started nibbling around the edges of opening some of this stuff up [3]. The Meta folks still aren't just handing out the big ones, but they're definitely moving the ball forward. In particular their release of the training logs is a really positive development IMHO as it opens the curtains a bit on the reality of training these things: it's a difficult, error/failure-prone process, there's a lo

Fortunately? Somebody already did the training for them! https://github.com/yandex/YaLM-100B

Direct link: https://github.com/yandex/YaLM-100B/blob/main/README.md#trai...

It appears the indexing for the model parts is deliberately not contiguous; the 03-82 range represents the main 80 transformer layers. https://github.com/yandex/YaLM-100B/blob/main/megatron_lm/me...

https://github.com/yandex/YaLM-100B/blob/main/LICENSE apache license >The model is published under the Apache 2.0 license that permits both research and commercial use you should be fine