PRE-TRAINING WITH CONTINUOUS CONCEPTS

Summary from the TLDR AI newsletter

CoCoMix is a pretraining framework that enhances next-token prediction by integrating continuous concepts learned from a sparse autoencoder. It improves sample efficiency, outperforming standard methods in language modeling and reasoning tasks. Additionally, it enhances interpretability by allowing direct inspection and modification of predicted concepts.