GPU-Puzzles

Oh hey, I wrote this. Been a long time. I had the lucky of break of working in machine translation / parsing when the most important invention of the century happened in my niche field. I'm pretty interested in the intersection of code / ML. If that's your thing here are some other writing you might be interested in. * Thinking about cuda: http://github.com/srush/gpu-puzzles * Tensors considered harmful: https://nlp.seas.harvard.edu/NamedTensor * Differentiating SVG: https://srush.github.io/DiffRast/ * Annotated S4: https://srush.github.io/annotated-s4/ Recently moved back to industry, so haven't had a chance to write in a while.

I think this might be good introduction to GPU programming: https://builds.modular.com/puzzles/introduction.html. It explains gpu concepts in an hardware agnostic way and verify understanding with implementation puzzle. It is based on https://github.com/srush/GPU-Puzzles but is CUDA specific.

These should keep you busy for months: https://www.gpumode.com/ resources and discord community Book: Programming massively parallel processors nvidia cuda docs are very comprehensive too https://github.com/srush/GPU-Puzzles

There are already some resources like that like: leetgpu.com https://github.com/srush/GPU-Puzzles For me its about a sense of progress, like in chess you can have an ELO score. Or in Duolingo theres a roadmap. If there were levels to this you could get more confident in your abilities. Right now the levels are basically bachelors, masters, and PhD. Coarse and expensive

If you're looking for a high level introduction to GPU development on Apple silicon I would recommend learning Metal. It's Apple's GPU acceleration language similar to CUDA for Nvidia hardware. I ported a set of puzzles for CUDA called GPU-Puzzles (a collection of exercises designed to teach GPU programming fundamentals)[1] to Metal [2]. I think it's a very accessible introduction to Metal and writing GPU kernels. [1] https://github.com/srush/GPU-Puzzles [2] https://github.com/abeleinin/Metal-Puzzles

Hello HN Community, I'm excited to share a project that's been brewing for a while now, aimed at GPU enthusiasts and developers alike. Say hello to GPU-Puzzles, an open-source repository that's set to revolutionize how we approach GPU programming and optimization. Why GPUs? In a world where tech giants like Google, Apple, and Facebook are pushing the boundaries of what's possible with AI, machine learning, and high-performance computing, the GPU has become the unsung hero. It's the powerhouse behind the scenes, and GPU-Puzzles is here to help you harness its full potential. What's inside? You'll find a curated collection of puzzles, challenges, and solutions that are designed to test and expand your understanding of GPU architecture and programming. Whether you're a seasoned developer or just starting out, there's something for everyone.

This: https://github.com/srush/GPU-Puzzles/raw/main/GPU_puzzlers_f...

Hey, I made these. They're pretty fun. Sometimes people tell me they use them for ML interviews, but they're kind of hard. The motivation was primarily teaching point-free, array programming. I don't think it is a great style, but it is fun as a brain teaser. If you enjoy this type of thing, I made a bunch more. They're all kind of ML + PL in style. - https://github.com/srush/gpu-puzzles - https://github.com/srush/tensor-puzzles - https://github.com/srush/autodiff-puzzles - https://github.com/srush/transformer-puzzles - https://github.com/srush/LLM-Training-Puzzles - https://github.com/srush/triton-puzzles All the graphics for these are made in Chalk which is a python port of Haskell's Diagrams library to https://github.com/chalk-diagrams/chalk . Honestly I mostly make the puzzles as an excuse to hack on the graphics library which I find

A great beginner guide to GPU programming concepts: https://github.com/srush/GPU-Puzzles

I really enjoy small elegant code demonstrations like this that really allow you to get your hands dirty to try to understand a concept. Another example is Sasha Rush's gpu puzzles, tensor puzzles -https://github.com/srush/GPU-Puzzles -https://github.com/srush/Tensor-Puzzles