PRE-TRAINING WITHOUT ATTENTION

Summary from the TLDR AI newsletter

Attention is a way for a language model to perform computation over words. It is pretty expensive. What if we could use 1d convolution state space models for BERT training? It turns out to perform quite well, all the code, weights, etc is available here in Jax.