ON EVALUATING AGENTS

Summary from the TLDR newsletter

No amount of evals will replace the need to look at the data. It's really difficult and time-intensive to evaluate agents' outputs when you are trying to validate complex conversation patterns. Stop relying solely on manual testing and create evals for agents. Tools help simplify things, but you still have to look at the data and build evaluations for your use case. Don't rely solely on standard evals - build your own.