sata_attention

The github repository's first toy example is with 8 Taylor terms, applied to a context with 1B tokens, each token with 1K heads: https://github.com/glassroom/sata_attention That toy example is not practical with the quadratic formulation, because it would require computing and storing 1K attention matrices, each with 1B×1B dot-product scores. For example, at Float32 precision, those attention matrices would consume approximately 1K x 1B x 1B x 4 bytes = 3,725,290,298.5 Terabytes of memory, which is not practical. Like every other proposed method, this one must be tested too. If it performs well in practice, AI service providers who ignore it will find themselves at a disadvantage. Otherwise, the mathematical techniques introduced by this work are likely useful for other applications besides Transformer attention.

Yes, there must be a connection. While adaptive truncation may prove impractical, it should be possible to measure spectral statistics on sample data, and specify a different fixed truncation order per layer, per head, etc. The github repository lists many other possible improvements: https://github.com/glassroom/sata_attention#proof-of-concept

Reference implementation: https://github.com/glassroom/sata_attention

Github here: https://github.com/glassroom/sata_attention