flaxformer

The add_zero_attn parameter in PyTorch is used for this, but by default their softmax is the regular kind. It has been in flaxformer for a couple years now though, however it claims to be a compatibility variant for older models [2] and I haven't seen any mention of it in their recent papers (though I've not checked exhaustively). [1]: https://pytorch.org/docs/stable/generated/torch.nn.Multihead... [2]: https://github.com/google/flaxformer/blob/main/flaxformer/co...

>why did no one come up with this before So it turns out someone did. Specifically google did. This exact same idea has been in flaxformers since at least November 2021. https://github.com/google/flaxformer/blame/ee62754ebe5a5eeb1... Specifically to save people a click it says: > """Softmax function with an additional virtual logit equal to zero. For compatibility with some previously trained models. This is equivalent to adding one to the denominator. In the context of attention, it allows you to attend to nothing. And creates the exact same modified softmax as this essay. I suppose only time will tell why it was ignored publicly before, maybe it doesn't do much, maybe it just fell through the cracks, maybe google just didnt push it, who knows

I might be missing something obvious, but I am not sure why everyone in the comments think it's a big deal. I've seen this trick in practice multiple times. For example, see this snippet from an old Google repo: https://github.com/google/flaxformer/blob/ee62754ebe5a5eeb11...