ARE AI LABS PELICANMAXXING?

Summary from the TLDR AI newsletter

Simon Willison's 'generate an SVG of a pelican riding a bicycle' benchmark has become one of the most famous informal benchmarks in AI. It is famous enough that there is a lot of discussion about its usefulness and whether AI labs might be making their models specifically better at the task. Testing has found that there is little evidence that AI labs are doing this. At least, not in a plainly obvious manner.