SIMPLEX RANDOM FEATURE ATTENTION

Summary from the TLDR AI newsletter

Attention is useful but computationally expensive. However, once trained, with a bit of fine-tuning compute you can drop in SRF attention and remove the dependence on sequence length, speeding things up considerably.