A NEW TRANSFORMER PASSES ITS HIDDEN STATE TO THE NEXT TOKEN INSTEAD OF RECOMPUTING IT

Summary from the TLDR AI newsletter

Researchers built LIFT, a transformer that feeds its internal state back into the next generation step instead of squeezing everything through the single token it outputs. Training stays parallel because the states come precomputed from an off-the-shelf model's next-token predictions, while at inference the model feeds back its own. LIFT models from 135M to 1B parameters beat token-matched standard transformers on language modeling, reasoning, and procedural tasks, so the next question is whether the gains hold at frontier scale.