WHY LANGUAGE MODELS HALLUCINATE

Summary from the TLDR Web Dev newsletter

This research paper from OpenAI investigates why language models like ChatGPT hallucinate. Current training and evaluation methods inadvertently reward guessing over admitting uncertainty, as accuracy-focused scoreboards prioritize correct answers, even if obtained through chance. This incentivizes models to guess instead of abstaining when unsure, leading to confident errors on low-frequency facts that are difficult to predict.