A CONTROLLED STUDY OF ATTENTION-ONLY TRANSFORMERS

Summary from the TLDR AI newsletter

The feed-forward network (FFN) has been a fixed component of the transformer block since its introduction. In modern decoder configurations, it holds roughly two-thirds of non-embedding parameters. A substantial interpretability literature argues that these layers act as the model's parametric memory. This experiment looks at what is actually lost when the FFN is removed entirely.