“IT'S HARD TO EVAL” IS A PRODUCT SMELL

Summary from the TLDR newsletter

Evals thinking is aligned with good product design. Gathering sporting data and breaking down workflows into smaller units makes automated grading easier. This post covers well-established design principles through the lens of the age of AI. Verification typically happens incidentally during the process of creating work product, but with AI, verification is the bottleneck.