gpt2-webgl

Here's a link to the github repo. At the top of the README it has a demo of GPT-2 running and the visualizations of the attention matrices and transformer block outputs Repo: https://github.com/nathan-barry/gpt2-webgl

CUDA is better for sure, but the pure functional nature of the traditional shader is conceptually much simpler and I kind of relish the simplicity. There is no crazy tiling or anything. Just per-pixel parallelism [1]. It won't be as fast as those real, highly-tuned kernels, but it's still nice to see something simple that does something non-trivial. It reminded me of the early "GPGPU" days (early 2000s?) [1] https://github.com/nathan-barry/gpt2-webgl/blob/main/src/gpt...

Yeah, right now the weights aren't even pushed to the repo so there's no way for the github page to load them in. I've looked a tiny bit into it and might end up making a fix for it, having it load the weights by fetching them from somewhere else on load, but probably not today. There is a link to the zipped up folder with the weights here: https://github.com/nathan-barry/gpt2-webgl/releases/download... If anyone wants to make a PR which fetches the file from there, unzips it, then loads in the weights from there that would be greatly appreciated.

Back in the old days, people used to do general-purpose GPU programming by using shaders like GLSL. This is what inspired NVIDIA (and other companies) to eventually create CUDA (and friends). This is an implementation of GPT-2 using WebGL and shaders. Enjoy!