BENCHMARK SCORES = GENERAL CAPABILITY + CLAUDINESS

Summary from the TLDR AI newsletter

In a 'deep' world, there is a single underlying ability that governs how well models do at superficially unrelated tasks. If a model developer makes this ability go up, their model gets better at everything. In a 'contingent' world, there are many orthogonal abilities that models can have, so model developers have to do completely unrelated work to get a model to improve on each ability. Anthropic has focused on making models that are state-of-the-art at agentic coding, but this hasn't resulted in models that are exceptional in other areas. There is some generalization across tasks, but this is limited, suggesting that models live in a 'contingent' world.