ROBOTS STRUGGLE TO COMBINE FAMILIAR INSTRUCTIONS IN NEW WAYS, HERE'S WHY

Summary from the TLDR Hardware newsletter

Robots trained on step-by-step instructions often fail on new combinations of familiar steps they weren't directly shown, and collecting demonstrations for every possible combination is impractically expensive. This paper breaks that failure down into three distinct causes - shifts in individual instructions, shifts in how instructions combine, and shifts in the surrounding context - to figure out exactly what a training set needs to cover rather than how much of it. Full coverage of every instruction combination isn't necessary: a carefully chosen quarter of the task space can match it, as long as that subset preserves the real dependencies between actions rather than just varying surface details.