AI Benchmarking Scandal: Did Big Tech Rig Chatbot Arena?

A new paper from Cohere, Stanford, MIT, and Ai2 accuses LM Arena, the organization behind the popular Chatbot Arena benchmark, of unfairly favoring top AI companies like Meta, OpenAI, Google, and Amazon. The researchers allege that these companies were allowed to privately test multiple model variants, suppressing poor-performing results to boost their leaderboard rankings. Analyzing over 2.8 million battles, the study found evidence of increased sampling rates giving these companies an unfair advantage. LM Arena disputes the findings, citing inaccuracies, and plans to improve its sampling algorithm, but denies manipulating rankings. The controversy raises concerns about fairness and transparency in AI benchmarking and highlights the competitive tactics employed by large tech companies in the AI race.