RESIDUAL CONTEXT DIFFUSION LANGUAGE MODELS

Summary from the TLDR AI newsletter

State-of-the-art block-wise Diffusion Large Language Models (dLLMs) rely on a remasking mechanism that decodes only the most confident tokens and discards the rest. Recycling computation from the discarded tokens is beneficial, as these tokens retain contextual information useful for subsequent decoding iterations. Residual Context Diffusion is a module that converts these discarded token representations into contextual residuals and injects them back for the next denoising step. It consistently improves frontier dLLMs in terms of accuracy with minimal extra computation overhead across a wide range of benchmarks.