THE AI INDUSTRY IS OBSESSED WITH CHATBOT ARENA, BUT IT MIGHT NOT BE THE BEST BENCHMARK

Summary from the TLDR newsletter

LMSYS' Chatbot Arena has become an industry obsession over the past few months. It lets anyone on the web ask questions of two randomly selected, anonymous models and then vote on their preferred answers. Critics say that LMSYS has not been completely transparent about the model capabilities, knowledge, and skills it's assessing on Chatbot Arena. The limited data released by the company makes it challenging to study the limitations of models in depth. While Chatbot Arena is framed as an empirical test, it amounts to a relative rating of models.