RECENT AI MODEL PROGRESS FEELS MOSTLY LIKE BULLSHIT

Summary from the TLDR Web Dev newsletter

A company using AI models for security vulnerability detection found that recent AI model improvements beyond Claude 3.5-3.7 Sonnet have not improved much on its internal benchmarks or in developers' ability to find new bugs. This goes to show that current benchmarks may not accurately reflect real-world usefulness and generality, with AI labs probably overfitting to specific benchmarks.