IMPROVING LLM CALIBRATION WITH LABEL SMOOTHING

Summary from the TLDR AI newsletter

Instruction tuning degrades confidence calibration in LLMs, making outputs less reliable. This paper proposes label smoothing as a remedy and introduces a memory-efficient loss kernel, showing that calibration issues grow with vocabulary size and hidden dimensions.