CHEAP RL TASKS WILL WASTE COMPUTE

Summary from the TLDR AI newsletter

AI labs will soon favor quality over quantity when procuring RL environments and spend a surprisingly large amount of money on procurement per RL task. Procuring RL tasks cheaply results in lower quality training runs, and the compute cost per RL run soon becomes very high. AI labs will increasingly move away from training on procedurally generated RL tasks and tasks produced cheaply by massive numbers of low-paid contractors toward a more labor-intensive build process that relies on full-time domain experts who spend months developing sustained context and crafting individualized tasks. The winners will be those who deliver the highest-quality, context-rich tasks, ship them rapidly, and price them efficiently given compute tasks.