DECOMPOSING LANGUAGE MODELS INTO UNDERSTANDABLE COMPONENTS

Summary from the TLDR newsletter

Neural networks are trained on data and not programmed to follow rules. While we understand the math of trained networks, we don't understand why those mathematical operations result in the behaviors we see, making it hard to diagnose failure modes and certify that a model is truly safe. This article presents a method for analyzing language models to determine how they work that involves looking at units in small transformer models rather than at individual neurons.