IT PASSED CI. IT PASSED YOUR EVALS. THE CUSTOMER STILL GOT THE WRONG ANSWER

Summary from the TLDR Data newsletter

AI agents can return successful, well-grounded responses that are still wrong because they retrieved invalid context or followed a faulty execution path. Debugging requires end-to-end traces of model calls, tool calls, retrieval inputs, and outputs, plus separate tests for deterministic system behavior and answer quality.