IMPROVED REPRESENTATION IN LANGUAGE MODELS

Summary from the TLDR AI newsletter

Masked mixers is a novel architecture that replaces self-attention with masked convolutions to improve input representation in language models.