llama2.c

Personal fave: https://github.com/karpathy/llama2.c

Llama2 inference can be implemented in 900-ish lines of dependency-free C89, with no code golfing[1]. More modern architectures (at least the dense, non-MoE models) aren't that much more complicated. That code is CPU only, uses float32 everywhere and doesn't do any optimizations, so it's not realistically usable for models beyond 100m params, but that's how much it takes to run the core algorithm. [1] https://github.com/karpathy/llama2.c

Links for llama2.c: https://github.com/karpathy/llama2.c https://news.ycombinator.com/item?id=36838051

What's really cool, is that this was built on Karpathy's llama2.c repo https://github.com/karpathy/llama2.c A great example of the unexpected things that happen when put great code into the commons. I bet Andrej never expected anything like this when he released it.

FOr inference, less than 1KLOC of pure, dependency-free C is enough (if you include the tokenizer and command line parsing)[1]. This was a non-obvious fact for me, in principle, you could run a modern LLM 20 years ago with just 1000 lines of code, assuming you're fine with things potentially taking days to run of course. Training wouldn't be that much harder, Micrograd[2] is 200LOC of pure Python, 1000 lines would probably be enough for training an (extremely slow) LLM. By "extremely slow", I mean that a training run that normally takes hours could probably take dozens of years, but the results would, in principle, be the same. If you were writing in C instead of Python and used something like Llama CPP's optimization tricks, you could probably get somewhat acceptable training performance in 2 or 3 KLOC. You'd still be off by one or two o

> Karpathy's TinyStories Did you mean Karpathy's tinyllamas? [1][2] Or did you mean Ronen Eldan and Yuanzhi Li's "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?" [3][4] [1]: https://huggingface.co/karpathy/tinyllamas/tree/main [2]: https://github.com/karpathy/llama2.c [3]: https://www.microsoft.com/en-us/research/publication/tinysto... [4]: https://arxiv.org/abs/2305.07759

I didn't realise llama.c was from him too. https://github.com/karpathy/llama2.c

On most hardware, handwritten math is required for all the nonstandard formats, e.g. for quantized int-8 https://github.com/karpathy/llama2.c/blob/master/runq.c#L317 Integer quantization doesn't typically just round, it has scaling and other factors in blocks so it's not just a question of manipulating int8's. And the FP16/FP8 are not supported by most processors so need their own custom routines as well. It would be great if you could just write code that operates with intrinsics on the quanitzed types.

Tiny Stories trained models and alternative implementation based on llama 2: https://github.com/karpathy/llama2.c

Related. I built karpathy’s llama2.c (https://github.com/karpathy/llama2.c) without modifications to WASM and run it in the browser. It was a fun exercise to directly compare native vs. Web perf. Getting 80% of native performance on my M1 Macbook Air and haven’t spent anytime optimizing the WASM side. Demo: https://diegomarcos.com/llama2.c-web/ Code: https://github.com/dmarcos/llama2.c-web

Would be cool for there to be a `llama2.f`, like https://github.com/karpathy/llama2.c, to demo its capabilities

This is a fork of https://github.com/karpathy/llama2.c karpathy's llama2.c is like llama.cpp but it is written in C and the python training code is available in that same repo. llama2.c's goal is to be a elegant single file C implementation of the inference and an elegant python implementation for training. His goal is for people to understand how llama 2 and LLM's work, so he keeps it simple and sweet. As the project progresses, so will features and performance improvements added. Currently it can infer baby (small) Story models trained by Karpathy at a fast pace. It can also infer Meta LLAMA 2 7b models, but at a very slow rate such as 1 token per second. So currently this can be used for learning or as a tech preview. Our friendly fork tries to make it portable, performant and more usable (bells and whistles) over time. Since we mirror

As a sibling comment mentioned, threading on a gpu is not automatically deterministic so you could randomness from there, although I can't think of anything in the forward pass of a normal LLM they would depend on execution order. So yes, you should get the same, it's basically just matrix multiplication. There may be some implementation details I don't know about that would add other variability though. Look at this minimal implementation (Karpathy's) of LLaMA, the only randomness is in the "sample" function that comes in at non-zero temperature, otherwise its easy to see everything is deterministic: https://github.com/karpathy/llama2.c/blob/master/run.c Otoh, with MoE like GPT-4 has, it can still vary at zero temperature.

I really enjoyed Anrej Kaparthy's llama2.c project (https://github.com/karpathy/llama2.c), which runs through creating and running a miniature Llama2 architecture model from scratch.

Have you seen the recent work at TinyStories: - https://arxiv.org/abs/2305.07759 It got some nice attention here: - https://github.com/karpathy/llama2.c I think there may be some applications in this limited space that are worth looking into. You won’t replicate GPT-anything but it may be possible to solve some nice problems very much more efficiently that one would expect at first.