STRUCTURED OUTPUTS CREATE FALSE CONFIDENCE

Summary from the TLDR AI newsletter

Constrained decoding forces models to prioritize output conformance over output quality. This is because it restricts models to only consider certain tokens. Using structured output makes it harder for models to refuse requests, warn users about contradictory information, and inform users of the right approach when they ask it to use the wrong approach. Letting models respond in as natural a fashion as possible is the most effective way to get the highest quality response from them.