THE ILLUSION OF DIMINISHING RETURNS: MEASURING LONG HORIZON EXECUTION IN LLMS

Summary from the TLDR AI newsletter

Real-world value often stems from the length of a task an agent can complete. Marginal gains in single-step accuracy can compound into exponential improvements in the length of a task a model can successfully complete. Models are more likely to make mistakes when the context contains errors from previous turns. Failures when tasks are made longer arise from mistakes in execution rather than an inability to reason.