flash-attention

1) Pretty much, it's mathematically equivalent. The only software issues are things like managing dependency versions and data formats in-memory, but Flash Attention 2 is already built into HuggingFace and other popular libraries. Flash Attention 3 probably will be soon, although it requires an H100 GPU to run 2) Flash Attention 2 added support for GQA in past version updates: https://github.com/Dao-AILab/flash-attention 3) They're comparing this implementation of Flash Attention (which is written in raw CUDA C++) to the Triton implementation of a similar algorithm (which is written in Triton): https://triton-lang.org/main/getting-started/tutorials/06-fu...

Two reasons (may not be well thought out or wrong): * Flash Attention, an efficient attention module which significantly speeds up training, only works on Ampere GPUs [1] * Even if I bought a 3090, I would have to get a computer to go with it, along with a PSU and some cooling. Don't know where to start with that. [1] https://github.com/Dao-AILab/flash-attention/issues/190