WHY DO DLLMS TEND TO COLLAPSE IN RL

Summary from the TLDR AI newsletter

Diffusion Language Models (dLLMs) experience training collapse during Reinforcement Learning because their log-likelihood must be estimated using high-variance Monte Carlo sampling, which creates noisy importance ratios. These noisy ratios induce gradient spikes that push policy drift in a positive feedback loop, a problem that traditional AR methods like conditional clipping fail to solve. The newly proposed StableDRL framework stabilizes the update process by combining unconditional clipping to suppress extreme values with self-normalization tied to the effective information in the batch.