tiny-cuda-nn

I really hope those pow(x, 2) calls are getting turned into x * x, else it's a performance catastrophe / extreme beginner mistake even with vectorisation. Also, this kind of ultra wide buffering consumes a ton of memory bandwidth for each operation, instead of keeping a small portion in cache/registers. FLOPs are scaling sort of infinitely, whereas memory speed is flat, so this is increasingly a losing game; just because it's faster than glacial Python doesn't mean it's fast compared to a language which actually concerns itself with performance or a more cache aware approach. For an extreme example of how you can even sometimes beat ultra optimised GPU ML libraries in this way, check out https://github.com/NVlabs/tiny-cuda-nn

...a C++ library with a CUDA backend. But these high-performance building blocks might only be saturating the GPU fully if the data is large enough. I haven't looked at implementing these things, but I imagine uf you have smaller networks and thus less data, the large building blocks may not be optimal. You may for example want to fuse some operations to reduce memory latency from repeated memory access. In PyTorch world, there are approaches for small networks as well, there is https://github.com/NVlabs/tiny-cuda-nn - as far as I understand from the first link in the README, it makes clever use of the CUDA shared memory, which can hold all the weights of a tiny network (but not larger ones).

The authors also provided demos here: https://tom94.net/data/publications/mueller21realtime/intera... Code: https://github.com/nvlabs/tiny-cuda-nn