SHAPING CAPABILITIES WITH TOKEN-LEVEL DATA FILTERING

Summary from the TLDR AI newsletter

Filtering pretraining data is a highly effective, robust, and inexpensive-at-scale way to reduce undesired capabilities in language models. Filtering tokens is more effective than filtering documents. Filtering gets more effective with scale. It is robust to noisy labels with sufficient pretraining compute.