llm.c

I ported Karparthy's llm.c repo to AMD devices [1], and have trained GPT2 from scratch with 10B tokens of fineweb-edu on a 4x 7900XTX machine in just a few hours (about $2 worth of electricity) [2]. I've also trained the larger GPT2-XL model from scratch on bigger CDNA machines. Works fine. [1] https://github.com/anthonix/llm.c [2] https://x.com/zealandic1

Hi, why do you believe that bfloat16 is not supported? Can you please provide some references (specifically the part about the hardware "doesn't do it")? For the hardware you are focussing on (gfx11), the reference manual [2] and the list of LLVM gfx11 instructions supported [1] describe the bfloat16 vdot & WMMA operations, and these are in fact implemented and working in various software such as composable kernels and rocBLAS, which I have used (and can guarantee they are not simply being run as float). I've also used these in the AMD fork of llm.c [3] Outside of gfx11, I have also used bfloat16 in CDNA2 & 3 devices, and they are working and being supported. Regarding cublasLt, what is your plan for support there? Pass everything through to hipblasLt (hipify style) or something else? Cheers, -A [1] https://llvm.org/docs/AMDGPU/AMDGPUAsmG

Yeah, I just reproduced the GPT2 from scratch results in 8.75 hours on 4x 7900 XTX. The fork is here: https://github.com/anthonix/llm.c

So... successfully reproduced in ~8.75 hours, taking about 18 kWh / $2.70 The first run actually failed at step 3000 or so, and I realized I had a bug in my attention / matmul kernels, but after fixing that and restarting it worked great [1] https://github.com/anthonix/llm.c

Seems to be an issue on their side. E.g., for a step of GPT2 training on a 7900 XTX [1]: tinygrad is ~440ms, PyTorch 2.4.0.dev20240513 is ~97ms, Karpathy's llm.c with ROCm is ~79ms, and llm.c with custom kernels is ~58ms [1] https://github.com/anthonix/llm.c [2] https://github.com/tinygrad/tinygrad/issues/4301