ml-engineering

For kernel-level performance tuning you can use the occupancy calculator as pointed out by jplusqualt or you can profile your kernel with Nsight compute which will give you a ton of info. But for model-wide performance, you basically have to come up with your own calculation to estimate the FLOPs required by your model and based on that figure out how well your model is maxing out the GPU capabilities (MFU/HFU). Here is a more in-depth example on how you might do this: https://github.com/stas00/ml-engineering/tree/master/trainin...

For anyone interested in tracking max achievable matmul FLOPS for hardware and unaware, I highly recommend tracking Stas Bekman's mamf-finder results: https://github.com/stas00/ml-engineering/tree/master/compute...

There is a lot of work to make the actual infrastructure and lower level management of lots and lots of GPUs/TPUs open as well - my team focuses on making the infrastructure bit at least a bit more approachable on GKE and Kubernetes. https://github.com/GoogleCloudPlatform/ai-on-gke/tree/main and https://github.com/google/xpk (a bit more focused on HPC, but includes AI) and https://github.com/stas00/ml-engineering (not associated with GKE, but describes training with SLURM) The actual training is still a bit of a small pool of very experienced people, but it's getting better. And every day serving models gets that much faster - you can often simply draft on Triton and TensorRT-LLM or vLLM and see significant wins month to month.

OK, the pdf is ready now: https://github.com/stas00/ml-engineering#pdf-version

I really appreciate everything in the "Unsolicited Advice" in the AI Battlefield section [1]. It's a very realistic take on the frenetic pace of everything and the emotional tax that comes with feeling like one is always drowning in the relentlessly rapid advance of AI development. Thanks! [1] https://github.com/stas00/ml-engineering/blob/master/insight...