HOW DO WE QUANTIFY PROGRESS IN ROBOTICS?

Summary from the TLDR newsletter

Offline data sets don't work because robots never do exactly the same thing. These errors compound. Robots are hardware plus an algorithm, and hardware factors often matter more to task execution than the algorithm being used. Any real benchmark by necessity should look like a human competition. There are already early versions of such events, like the World Humanoid Games. However, major companies right now have too much to lose and too little to gain to bother competing.