nanoGPT

> So, why do people keep making these? Asking genuinely I extensively used minGPT for home experiments on transformer architecture. It is great for learning! However, if you want to scale the experiments up at home you need to go faster. Karpathy made optimized https://github.com/karpathy/nanoGPT, but it is tuned for "8XA100 40GB node in about 4 days of training". 13s is a bit overkill here (my machine builds that project in 30s). But it gives some space for experimentation with architectures that don't have optimized primitives.

This weekend I just cracked into nanoGPT (https://github.com/karpathy/nanoGPT), an older but fabulous learning exercise where you build and train a crappy shakespeare GPT with ~0.8M parameters on a cpu. Results are about what you'd expect from that, they suck, but you can start to feel the magic, especially if you're not a deep learning professional and you just want to poke around and hack on it. I started writing up a blog post on my weekend with nanoGPT but it's not done yet... Would have been great to link to here lol oh well

Andrej Karpathy's Nano GPT is reasonably accessible and easy to run. https://github.com/karpathy/nanoGPT

To expand on the other comment, if you look under the data folder in nanoGPT, you can see examples of how to train the model using various data sources and encoders. "shakespeare_char" is probably the most rudimentary, only converting the characters of the input into integers. e.g. https://github.com/karpathy/nanoGPT/blob/master/data/shakesp...

you got to be joking, do you play with slide rules and abacus too NN in Mathematica is a just a toy and artifact of a bygone era. Try HuggingFace https://github.com/huggingface/course or Karpathy's intro https://github.com/karpathy/nanoGPT

The one you linked to is based on Karpathy's tutorial: https://www.youtube.com/watch?v=kCc8FmEb1nY, except changing to train on tinystories in stead of Shakespeare. smolGPT also looks inspired by nanoGPT, also from Karpathy: https://github.com/karpathy/nanoGPT/blob/master/train.py

This isn't true. Learning how to train a 124M is just as useful as a 700B, and is possible on a laptop. https://github.com/karpathy/nanoGPT

Karpathy's nanoGPT has a full training pipeline using Shakespeare. [1] The use case for this is learning in simple example. [1] https://github.com/karpathy/nanoGPT

github has a bunch of them for years, the most known from Andrej Karpathy: https://github.com/karpathy/nanoGPT some other have MoE implemented.

I like hands-on learning more than I like textbooks, so in case that matches your requirements, maybe try training your own GPT to have a sense for how it works. I wrote a Rust version of the famous https://github.com/karpathy/nanoGPT (which is in Python) so that I could learn how it's built. I wrote it in Rust because I wanted to improve my skills in that language, be forced to write code instead of just reading the existing implementation so that I would truly learn, and test the quality of the nascent Rust AI/ML ecosystem, but you could pick your own language

You would have to train the new model from scratch since it would be all new token embeddings with whatever character encoding scheme you come up with. It would probably make sense to train the vanilla gpt from scratch with the same total embeddings size as your control. I would start with https://github.com/karpathy/nanoGPT as a baseline since you can train a toy (GPT2 sized) llm in a couple days on an a100 which are pretty easy to come by.

There are a bunch of good answers, but I wanted to succinctly say "practically, quite a bit". Here's a good little rabbit-hole example: > https://github.com/karpathy/nanoGPT/blob/master/model.py#L45 Karpathy's nanoGPT calling flash attention by checking if torch.nn.functional.scaled_dot_product_attention exists > https://pytorch.org/docs/stable/generated/torch.nn.functiona... Looking at the docs, in reality, most of the time you want this to call out to FA2 which optimizes the kernals on the device to split ops on the Softmax of the triangular matrix as well as reduce moving unnecessary batches of floating point numbers back and forth from the GPU to the CPU. > https://arxiv.org/pdf/2307.08691 The paper for FA2 almost entirely considers itself through the hardware it's running on.