SELF-ATTENTION WITH CONVOLUTIONS

Summary from the TLDR AI newsletter

Researchers have developed a novel approach called FFNification that transforms self-attention mechanisms into more efficient token mixers using only convolutions while keeping the query-key-value framework.