THE MARKET FOR LONG HORIZON TASKS

Summary from the TLDR Founders newsletter

There's a strange wedge hiding here, building the hard tasks labs use to test their agents. A convincing fake of SAP, realistic enough to evaluate an agent inside, was rumored at $500,000, and METR says it can barely find long-horizon tasks good enough to use, with decent ones at $20,000 each. Cheap labeling is everywhere, but captured expert work that's realistic enough to fool an agent and clean enough to grade is scarce, so the eval bottleneck is domain realism, not model access.