BERT IS JUST A SINGLE TEXT DIFFUSION STEP

Summary from the TLDR AI newsletter

BERT-style models are essentially just text diffusion models trained on one masking rate. Masked language models can be repurposed into full generative engines by interpreting variable-rate masking as a discrete diffusion process. The standard masked language modeling objective can be turned into a step-by-step generation procedure by gradually corrupting text with mask tokens and training the model to iteratively denoise at increasing mask intensities. A fine-tuned RoBERTa can generate coherent-looking text after slightly modifying the training objective, even without architectural changes.