Are the leaderboards lying to you?

Are the leaderboards lying to you? That is the question for this episode, and I want to answer it the way we always do: by reading the studies instead of the headlines.

Here is the setup. LMArena, formerly Chatbot Arena, is the most cited model leaderboard in the world. It works like a blind taste test. Two anonymous models answer the same prompt, a human votes for the better answer, and the votes get converted into an Elo rating, the same math chess uses. Roughly two million of those battles feed the rankings. When someone tells you which AI is best, this chart is usually where the claim comes from.

In 2025 a team led by Singh published "The Leaderboard Illusion," peer reviewed at NeurIPS 2025. Funding flag, said out loud like always: several authors are affiliated with Cohere, a company whose models compete on that same leaderboard, alongside academic co-authors from Princeton, Stanford, and MIT. Cohere has a stake in the result, so keep that in your head the whole way through.

Here is what the paper documented. First, private testing. Large labs can quietly submit many variants of an unreleased model, watch the scores, and publish only the winner. The paper counts Meta testing 27 private variants of Llama 4 before release. Statisticians call this best-of-N gaming: enter enough versions and one will score high by luck alone, then you show the world only that one. It is the same selection effect as a drug company running ten trials and publishing the one that worked. Elo math assumes each entry is one honest player; best-of-N quietly breaks that assumption. Second, selective retraction: disappointing scores could be pulled before publication. Third, the arena's own data: up to 26.5 percent of prompts were duplicates or near duplicates, which rewards anyone who has already seen the traffic.

Then the paper's sharpest experiment. The authors fine-tuned a model on arena-style data, and its win rate on ArenaHard jumped 112 percent. Its score on MMLU, a broad knowledge test, gained nothing and actually slipped slightly. Read that back. A model climbed the board dramatically while getting no smarter by any other measure. That is a leaderboard measuring something other than capability.

Now the other side, because this one is genuinely contested, and our standard says contested claims get both sides with no winner declared. LMArena published a public rebuttal disputing parts of the paper's framing and its characterization of the arena's policies. You should read both documents, and you should flag both parties while you do. The critics are affiliated with a competitor on the board. The defender runs the board, and LMArena is now VC backed, which means the referee has investors. Nobody in this fight is disinterested.

There is also supporting science from groups outside the fight. One 2025 preprint, independent, showed that dropping just a handful of preference votes can change which model ranks first. Another independent 2025 study showed rankings can be deliberately rigged with a few hundred injected votes out of roughly 1.7 million. So even with nobody gaming anything, small gaps at the top of an Elo board sit inside the noise.

Time for the graded verdict. The mechanism is strong. Private variant testing is documented, the fine-tuning experiment that moved the ranking without moving capability is checkable, and the vote-fragility results confirm from an independent angle that top rankings are brittle. The intent question, whether LMArena's policies amount to a rigged game or a defensible platform under commercial pressure, is contested, and we present both sides and stop there.

What should you do with this? Treat any leaderboard rank as a claim, and check who submitted what before you repeat it. When two models sit within a few points of each other, call it a tie. Whitney is next with the stranger half of the story: why the same model's score moves even when nobody is gaming anything.

Sources

• Singh et al., "The Leaderboard Illusion" (NeurIPS 2025, peer reviewed; preprint arXiv:2504.20879, April 2025) [PR][Mixed: Cohere-affiliated with Princeton/Stanford/MIT co-authors] - https://arxiv.org/abs/2504.20879
• LMArena public rebuttal to "The Leaderboard Illusion" (2025) [Vendor: LMArena, VC-backed]
• "Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings" (arXiv, 2025) [Independent] - https://arxiv.org/pdf/2508.11847
• "Improving Your Model Ranking on Chatbot Arena by Vote Rigging" (arXiv:2501.17858, 2025) [Independent] - https://arxiv.org/html/2501.17858
• The AI Inevitable Evidence Standard v1.1 (evaluator table, LMArena caveat) - docs/research/evidence-standard.md

Why does a model win on Tuesday and lose on Thursday? You run a comparison, pick a winner, and a week later the same two models rank the other way around in somebody else's test. The weights never changed. So what moved?

Michael just showed you how a leaderboard can be gamed on purpose. I want to show you the stranger result: scores that churn with nobody cheating at all.

Start with the biggest number in this episode. A 2026 study, an arXiv preprint, measured how much of the variation in benchmark results comes from the model itself versus the harness, meaning the software wrapper that feeds the model its prompts, tools, and context. Harness variance came out 7.8 times larger than model variance. Let me say that slowly. The wrapper around the model moved the scores almost eight times more than swapping the model did. Our show re-checked that figure against the primary source in July 2026, and it held.

The scaffold data points the same direction. On agentic benchmarks, identical weights swing 10 to 35 points depending on the scaffold alone. The weights did not change. The scaffold did, and the score moved by double digits.

METR, an independent evaluation nonprofit, adds a wrinkle in a February 2026 research note: popular agent scaffolds like Claude Code and Codex are not obviously better than the plain default scaffolds on autonomous tasks. Funding flag: METR has had Anthropic-affiliated authors and funding ties. Its methods are published, so you can check the work, but it is not disinterested.

Why is the harness so heavy? Because almost everything you experience as the model's behavior passes through it. The system prompt. The tool definitions. How context gets assembled. Sampling defaults, like how much reasoning effort is switched on. Community-maintained prompt archives document that Cursor, GitHub Copilot, and Claude Code each ship their own substantial system prompt and their own tool definitions around identical weights. Anthropic's own documentation, a vendor source, so flag it, confirms the weights are the same across access surfaces while the feature sets and defaults differ. Same engine, different car.

And underneath all of that sits a floor of pure noise. The same model returns different outputs for the same prompt even at temperature zero, the setting that is supposed to be deterministic. A September 2025 technical investigation from Thinking Machines Lab, a company publishing its own work, so flag that too, though the code is released and reproducible, ran one prompt a thousand times against one endpoint and got roughly 80 distinct outputs. The cause is a lack of batch invariance: the numerical kernels give slightly different results depending on how many other requests share the server batch, and batch size depends on server load at that instant, which no caller controls. Two calls to the same endpoint on the same day can diverge. A single Tuesday-versus-Thursday comparison cannot even distinguish a harness effect from serving jitter.

Put it all together and you get the rule our published Evidence Standard states plainly: a benchmark score is a property of the model plus the scaffold plus the test-time compute plus the harness plus the data subset. Change any one of those and the score moves. Two runs with different harnesses are measuring two different systems that happen to share weights. The Tuesday model and the Thursday model were never the same test subject.

Verdict: the evidence here is strong. The 7.8 times figure comes from one study, so hold that one number with care. But the scaffold swings, the prompt archives, the METR note, and the nondeterminism experiment come from separate groups with different funders, and they converge on the same conclusion: the harness carries more of the score than the model swap does.

So what should you do? When a score changes, ask what else changed before you credit or blame the model: the harness, the scaffold, the data subset, the sampling settings. Run any comparison you care about more than once, because a single trial cannot beat the noise floor. And when a published comparison does not tell you who ran the harness and how it was configured, assume the harness explains the difference until someone shows you otherwise.

Sources

• Harness-variance study (2026) [arXiv] - harness variance 7.8x model variance; figure re-verified against the primary source in the Evidence Standard's July 2026 three-pass check - docs/research/evidence-standard.md
• Scaffold swings of 10 to 35 points on agentic benchmarks by scaffold alone - The AI Inevitable Evidence Standard v1.1, third-party verification section - docs/research/evidence-standard.md
• "Measuring Time Horizon using Claude Code and Codex" - METR research note, Feb 13, 2026 [Independent nonprofit; Anthropic-affiliated author/funding ties flagged] - https://metr.org/notes/2026-02-13-measuring-time-horizon-using-claude-code-and-codex/
• Horace He & Thinking Machines Lab, "Defeating Nondeterminism in LLM Inference" (Sept 10, 2025) [Company technical investigation; reproducible code released] - https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
• system_prompts_leaks - asgeirtj (community-maintained system-prompt archive) - https://github.com/asgeirtj/system_prompts_leaks
• Claude on Amazon Bedrock - Anthropic API documentation [Vendor documentation] - https://platform.claude.com/docs/en/build-with-claude/claude-on-amazon-bedrock

How do you read a benchmark like a scientist? Whitney just showed you that a score belongs to a whole measurement system, and I opened the episode with a leaderboard that could be climbed without any gain in capability. This closing question is the method: four checks to run before you believe any number on any chart.

Check one: construct validity. Does the test measure what its name claims? The biggest study on this question is a NeurIPS 2025 systematic review from Bean, Kearns, and colleagues. Peer reviewed, independent. They put 445 language model benchmarks from top venues in front of 29 expert reviewers and found construct-validity problems nearly everywhere they looked. Benchmarks frequently do not measure the capability they claim to measure, and the team issued eight corrective recommendations, which tells you the problem is structural. Here is the consequence to carry home: a benchmark score does not license a claim about general capability. A model that tops a reasoning benchmark has demonstrated performance on that benchmark, full stop. An ICML 2025 position paper reached the same conclusion for medical benchmarks specifically. Peer reviewed, independent. And this critique has a paper trail: Raji and colleagues argued back in 2021 that framing benchmarks as measures of general capability is, in their words, dangerous and deceptive.

Check two: contamination and inflation. Could the model have seen the answers, and does the grader actually grade? SWE-bench, the flagship coding benchmark, is the case study. Wang, Pradel, and Liu, peer reviewed at ICSE 2026, independent, found that thin test coverage let about 7.8 percent of plausible-looking patches count as correct even though they failed the full developer test suite. Different 7.8 from Whitney's, a coincidence of digits. That inflated reported resolution rates by roughly 6.4 points. Then a separate independent team built UTBoost, strengthened the test suites, and enough verdicts flipped to affect 24.4 percent of SWE-bench Verified entries and produce 18 ranking changes. Stronger grading reordered the leaderboard. Separate diagnostic work traces a share of the passes to solution leakage and memorization rather than reasoning.

Check three: saturation. When every frontier model scores above roughly 88 percent on MMLU, the remaining gaps are statistically uninformative. That figure comes from a 2026 industry report by Kili Technology, so flag it: an industry source, directional rather than peer reviewed. A saturated test cannot separate the leaders, no matter how confidently the chart is drawn.

Check four: who runs the harness, and who paid? Ask this about the benchmark itself, and again about the paper citing it. Our published evaluator table reads like a disclosure sheet, because it is one. Epoch AI's FrontierMath benchmark turned out to be OpenAI funded, a disclosure lapse, and yet Epoch's fixed-harness rerun is also what caught OpenAI's o3 overclaim, cutting a reported 25 percent down to around 10. LMArena is now VC backed. Scale's SEAL leaderboard is vendor owned, and its private curation carries its own accountability gap. Even the referees have conflicts. That is why we flag every source, every time, including the ones we agree with.

Now the verdict for the whole episode, graded per our standard. The evidence is strong. Independent groups with different funders converge from different angles. A leaderboard was climbed without capability gains. Harnesses move scores far more than model swaps do. And the tests themselves often fail to measure what their names promise. Where the story is contested, the LMArena dispute, we presented both sides and declared no winner.

Here is the practical close for the whole episode. Use leaderboards to shortlist, never to decide. Let public scores cut the field to a few candidates, then run your own evaluation: 20 to 100 real tasks from your own workload, graded by your own pass criteria, repeated enough times to beat the noise floor Whitney described. Treat gaps under five points between frontier models as ties. And before you repeat any benchmark number, check who ran the harness and who paid for the test. If you cannot find out, treat the number as unverified and move on.

Sources

• Bean, Kearns, et al., "Measuring what Matters: Construct Validity in Large Language Model Benchmarks" (NeurIPS 2025 Datasets & Benchmarks; 445 benchmarks, 29 reviewers) [PR][Independent] - https://arxiv.org/abs/2511.04703
• ICML 2025 medical-benchmark position paper [PR][Independent] - per docs/research/question-pool.md, cluster B1
• Raji et al. (2021) and Bowman & Dahl (2021), via "Can We Trust AI Benchmarks? An Interdisciplinary Review" (arXiv, Feb 2025) [Independent] - https://arxiv.org/html/2502.06559v1
• Wang, Pradel, Liu, "Are 'Solved Issues' in SWE-bench Really Solved Correctly?" (ICSE 2026, peer reviewed; arXiv:2503.15223) [PR][Independent] - https://arxiv.org/abs/2503.15223
• Yu, Zhu, He, Kang, "UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench" (arXiv:2506.09289, June 2025) [Independent] - https://arxiv.org/abs/2506.09289
• SWE-bench Verified contamination/memorization summary (EmergentMind topic overview, 2025) - https://www.emergentmind.com/topics/swe-bench-verified-68233829-9998-47b4-a508-46198b489321
• "AI Benchmarks 2026: Top Evaluations and Their Limits" - Kili Technology (industry report; directional, flagged) - https://kili-technology.com/blog/ai-benchmarks-guide-the-top-evaluations-in-2026-and-why-theyre-not-enough
• The AI Inevitable Evidence Standard v1.1 (evaluator table: Epoch/FrontierMath, LMArena, Scale SEAL caveats) - docs/research/evidence-standard.md
• Singh et al., "The Leaderboard Illusion" (NeurIPS 2025) [PR][Mixed: Cohere-affiliated] - https://arxiv.org/abs/2504.20879