ASKING LLM FOR CONFIDENCE

Summary from the TLDR Dev newsletter

An LLM's stated confidence is generated text, so a precise-looking number should not be treated as a calibrated measure of correctness. The proposed alternative derives scores from observable evidence and historical outcomes, then checks calibration regularly on the same kinds of tasks.