REASONING

Summary from the TLDR AI newsletter

The recent work on registers for ViTs showed that adding a [reg] token fixes image attention map spikes. This work shows that adding a series of learnable [pause] tokens prepended to the sequence improves language model reasoning (decoder only, up to 1B) by as much as 18%. The intuition is that this adds a number of learnable and manipulatable vectors to store important global information to be used while generating the sequence. In general, adding tokens has scaled well past 13B parameters, but it's unclear how well this works at frontier model scale.