TLDR · PROGRAMMING, DESIGN & DATA SCIENCE

HOW WELL DO AGENTS USE TEST/VERIFICATION TECHNIQUES?

danluu.com68 min read

Summary from the TLDR newsletter

It's easier than ever to hit a particular quality bar by having coding agents use effective test techniques. However, software quality seems to be getting worse. This indicates that whatever defaults developers are using might not work very well. This study tests whether simple instructions to agents to use particular techniques or libraries improve implementation correctness. It found that agents tend to either just write the tests they would normally write, but inside a framework for a different type of test technique, or they'll use a technique superficially but not really do the things that get the value out of the technique.