Contamination Is the Least of It — Why Harder and More Private Benchmarks Won't Fix AI Measurement

The Benchmark That Marked Right Answers Wrong

OpenAI went looking for the limits of its own model. It pulled 138 tasks from SWE-bench Verified that GPT-5.2 had repeatedly failed and audited them one at a time, expecting a map of what the model could not yet do [2].

The map it got was of the benchmark. 59.4% of the audited tasks were flawed: 35.5% turned on specific function names that never appeared in the prompt, and 18.8% checked for features copied in from unrelated pull requests [2]. The tasks were not failing because the model wrote bad code. Many were failing because the tests demanded a particular implementation and rejected every other correct one.

On 23 February 2026, OpenAI stopped evaluating models against SWE-bench Verified [3]. The switch is recorded independently in the research literature, which notes that OpenAI now recommends SWE-Bench Pro in place of SWE-Bench Verified, on the stated grounds that Verified "no longer measures frontier coding capabilities" [1].

The distance between the two benchmarks is not a rounding error. Claude Opus 4.5 scored 80.9% on Verified in March 2026 and around 23% on SWE-bench Pro [3]; models sitting at roughly 70% on Verified landed at roughly 23% on the Pro public split [2]. For two years, the industry read the Verified column as the state of frontier coding.

The audit figures are OpenAI's own account of its own testing, reported secondhand by trade coverage rather than published as a paper with the task list attached; the part of this that an outside party recorded independently is the abandonment and the reason given for it.

The tasks were not too easy — frontier models were failing them. The answers were not leaked — the tests were simply wrong. Something had gone wrong with the instrument that the standard vocabulary has no word for.

The Story We Already Tell

That vocabulary has two words in it, and both describe something real.

The first is saturation. In the one systematic study that has tried to measure it across the field, saturation is not "high scores" — it is defined as the loss of reliable discriminative power among state-of-the-art models [4]. The distinction matters: a benchmark is dead when it stops separating models, which can happen long before any model answers everything correctly. Of the 60 benchmarks that study analysed, 29 exhibit high or very high saturation at an S_index of 0.7 or above, and 14 fall into the very high band at 0.9 or above [4].

The second is data contamination: "the accidental or deliberate inclusion of evaluation or benchmark data in the training phase of language models, resulting in artificially high benchmark scores" [5]. The standard verbatim test is n-gram matching — OpenAI used 13-gram matches to detect overlaps between test sets and training data for GPT-3 [5] — on the reasoning that at thirteen tokens, accidental occurrence becomes vanishingly unlikely [6]. Contamination also has a paraphrased form, where the semantic content survives with the surface changed, and that form slips past string matching entirely [6].

The strongest evidence for this frame is a benchmark built to test it. GSM1k is a set of 1,250 grade-school maths problems written by human annotators sourced through Scale AI, with no language models used in its construction, mirroring the style and complexity of GSM8k [7]. If a GSM8k score measured arithmetic, the two scores should agree. For many models they did not: the arXiv version of the study reports performance drops of up to 13% accuracy [7], the peer-reviewed version reports drops of up to 8%, with several families of models showing evidence of systematic overfitting across almost all model sizes [8]. And the mechanism looks like exposure — the same paper finds a Spearman's r² of 0.36 between a model's probability of generating a GSM8k example and its performance gap between the two sets [8].

This is what overfitting to a benchmark means, and it is worth keeping distinct from contamination: a model can be tuned toward a benchmark's particular flavour of problem without the test items themselves ever entering training. Contamination is one route there. It is not the only one — the GSM1k authors say as much about their own result, in a section headed "Data Contamination Is Likely Not The Full Explanation for Overfitting" [7].

Nor is contamination itself a solved measurement problem. An analysis of four contamination-detection metrics across thirteen benchmarks and seven models finds that contamination has a larger impact than previously reported, that the gap is driven by false negatives in existing detection methods, and that larger models are better able to leverage contaminated examples — Llama 1 showed performance increases of over 20% on several benchmarks [9]. The detectors under-report, and they under-report most for exactly the models anyone cares about.

So the frame does real work. But notice its shape. Everyone reaches for the exam metaphor — the model saw the answer key — and that metaphor makes benchmark failure a security problem whose fix is secrecy. The SWE-bench audit is not that. A test suite that rejects correct code is a scale that was never zeroed: it reads wrong for everyone, honest and dishonest alike, and locking it in a drawer would not improve it by a single point. Contamination is one way an instrument goes wrong. It is not the category.

What That Story Cannot See

Those other ways begin with what a benchmark score names. It names a model; the number is produced by a model, a harness and an environment together — and coding benchmarks, designed in a pre-agent era, collapse all three into a single end-to-end score with no component-level signal for iteration [1].

How much the harness moves that score is measurable. On Terminal-Bench, agent harnesses running the same underlying model — Claude Opus 4.6 — produced accuracies ranging from 58.0% under Claude Code to 79.8% under ForgeCode [1]. A harness, here, is the scaffolding around the model: the tool loop, the file access, the retry policy, the prompt template that decides what the model sees and how many times it gets to try. Twenty-two points of a headline score belong to software that is not the model, and the leaderboard column says only the model's name.

The second break shows up in the grading. SWE-bench and its variants anchor on a single reference solution: the FAIL_TO_PASS and PASS_TO_PASS test sets are derived from the original pull request, so an agent is graded on whether its patch matches the reference implementation rather than on whether the bug is fixed [1]. In the authors' phrasing, "an agent that resolves a flaky test by restating the API at a different level of abstraction is judged not on whether the bug is fixed, but on whether the reference tests still hold" [1]. This is the same defect the OpenAI audit found, stated as a design property rather than as a discovery. A benchmark built this way does not have a contamination problem to solve. It has a definition problem.

The third break is attribution. Run GPT-5-nano inside an agentic loop on 100 samples from SWE-bench Verified and it resolves 31% of them; give the same model the same tasks as a long-context problem, with no loop, and it resolves 0% [10]. The model, the tasks and the grader did not change: thirty-one points belong to the scaffold. Whatever the SWE-bench column is measuring in the agentic setting, it is not the model in isolation.

And where the grader is itself a model, the instrument has preferences of its own. A systematic study of fifteen LLM judges drawn from the GPT, Claude, Gemini and Llama series documents position bias — a tendency to favour solutions by where they sit in the prompt rather than by what they contain — and shows it is systematic rather than random variation [11]. An open-source judge-reliability harness released by the RAND Corporation in March 2026 found no judge uniformly reliable across benchmarks, with frontier models exceeding 50% error rates on bias tests [12]. LiveBench's designers cite bias and error rates of up to 46% for judging hard problems as the reason the suite scores automatically against ground truth instead of using human or LLM judges at all [13].

None of this is an exposure problem. A private test set graded against a broken reference solution is still graded against a broken reference solution. A harness that adds twenty points adds them whether or not anyone can read the questions. A judge that prefers the first answer prefers it in secret as readily as in public. And these failures fall precisely on the agentic tasks the industry now buys models to perform.

What the Field Bought Instead

The field's answer was to keep the questions where the model cannot have seen them.

Harder. Humanity's Last Exam was assembled by nearly 1,000 subject-expert contributors from over 500 institutions across 50 countries, on the explicit reasoning that "LLMs now achieve over 90% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities" [14]. ARC-AGI-3 went further, into hundreds of interactive environments where humans score 100% and frontier AI scores 0.51%, with over $2,000,000 in prizes attached [15]; at launch, every frontier model tested scored below 1% on the semi-private evaluation set [16].

Fresher. LiveBench draws its tasks from recent competitions and news and is updated monthly to maintain contamination resistance [13]. One account of the Arena reports that about 75% of daily prompts are fresh, and offers that as contamination resistance [17].

Private. Scale's SEAL leaderboards rank models on curated private datasets kept unpublished specifically so they cannot be incorporated into training data [18]; the datasets hold roughly 1,000 examples each [19]. Microsoft AI reports running a suite of nearly 40 internal benchmarks focused on held-out negative-log-likelihood tasks, sourced from vendors who created data fully held out from training, from internal sources not present on the web, and from web sources it removes from its own training data [20]. And OpenAI, in the same announcement that abandoned Verified, said it would move to privately authored evaluations such as GDPVal to avoid leakage [2].

Every one of these is a defence against exposure, bought immediately after a failure that had nothing to do with exposure.

The Case That This Is the Correction Loop Working

MMLU saturates, and Humanity's Last Exam gets built [14]. GSM8k leaks, and GSM1k gets built — and Scale adopts it as the maths leaderboard on SEAL [19]. SWE-bench Verified turns out to reject correct code, and the lab most dependent on it drops the benchmark within weeks of its own audit [3], with the replacement recorded in the literature [1]. The Arena is gamed, and a paper from Cohere Labs, AI2, Princeton, Stanford, Waterloo and Washington says so in public [21]. ARC-AGI-3 restores an enormous gap on purpose, measuring skill-acquisition efficiency rather than crystallised knowledge [16]. That is what a functioning field looks like from the inside: instruments fail, the failures get published, better instruments get built, and the complaint that measurement is broken is really an observation that measurement is being repaired in public.

The evidence concedes something more specific, too, and it comes from the same GSM1k study that supplies the frame's best evidence. Alongside the drops, "many models, especially those on the frontier, show minimal signs of overfitting, and all models broadly demonstrate generalization to novel math problems guaranteed to not be in their training data" [8].

On problems constructed so that no model could have seen them, every model still did the arithmetic. If benchmark numbers were pure memorisation artefacts, that would not happen.

Why the Loop Buys Time, Not Measurement

The loop is real. What it produces is a new benchmark, and the case for the loop assumes that a new benchmark is a working instrument.

The exposure model the remedies assume is not supported by the one study that tested it. If contamination is what wears benchmarks out, private held-out benchmarks should decay more slowly than public ones. The systematic study of 60 benchmarks tested exactly that and found no statistically meaningful difference in saturation indices between public benchmarks and private ones with held-out test sets [4]. What predicted saturation instead was age, adoption and test-set size, with larger test sets associated with lower saturation — measurement resolution, not leakage [4]. One study of 60 benchmarks is not a settled literature. But it is the only systematic test of the premise the entire private-benchmark movement rests on, and it comes back negative.

Privacy relocates the trust problem rather than resolving it. A public benchmark asks you to trust a dataset you can inspect. A private one asks you to trust an institution you cannot. FrontierMath is the clean case: OpenAI commissioned the production of 300 questions, fully owns them, and has access to all statements and solutions except a subset of 50 solutions randomly withheld for holdout evaluation [22]. OpenAI's o3 announcement claimed 25.2% on the benchmark [22]. Epoch AI's funding by OpenAI was not disclosed until December 2024; its co-founder acknowledged the organisation had "made a mistake" on transparency, and its lead mathematician said at the time that Epoch had not yet independently verified OpenAI's scores [23].

The Arena shows the same structure from the other direction. Its scores are relative estimates from a Bradley-Terry model rendered on an Elo-like scale, so a rating describes performance inside a particular voting pool rather than absolute quality [17]. Undisclosed private testing lets a handful of providers try many variants and disclose only the best: 27 private LLM variants were tested by Meta in the lead-up to the Llama-4 release, and 205 of 243 public models have been silently deprecated [21]. One analysis estimates that modest increases in access to Arena data could boost a model's Arena performance by up to 112% [24], and reads the episode as Goodhart's law with a leaderboard attached — "when a measure becomes a target, it ceases to be a good measure" [24]. Neither the private benchmark nor the live one is disciplined by the property that made public static benchmarks checkable: a stranger being able to look.

Difficulty has a half-life, and it never conferred validity in the first place. Humanity's Last Exam's own page allowed that "it is plausible that models could exceed 50% accuracy on HLE by the end of 2025" [14]. The trajectory since is steep even where the prediction was early: the exam's own reporting has Gemini 3 Pro at 38.3% [14], while one public tracker puts the top score at 40.0%, held by Claude Opus 4.6 [25], and another, updated 20 July 2026, puts its leader at 0.647 — a figure it reports as a fraction rather than a percentage [26]. Those two trackers cover different model sets and cannot be reconciled with each other; that they disagree by that much about the same exam is itself an instrument problem, and neither figure should be read as a trend line.

The deeper point is that difficulty was never the property in question. SWE-Bench Verified curated 500 human-validated tasks [1] — human validation being precisely the quality that was supposed to make it trustworthy — and it was hard enough that frontier models kept failing it. It was also rejecting correct code. Neither difficulty nor freshness nor privacy would have caught that, because none of them is a check on whether a test measures what its name says.

What Measuring Would Actually Cost

Strip away the remedies and one requirement is left. A number earns trust when someone has argued that it measures the construct it is named after, and someone with no stake in the answer can check that argument. Both halves are expensive, and the field has been paying for neither.

There is vocabulary for the first half, borrowed from psychological testing. A 2025 ICML position paper on medical LLM benchmarks defines construct validity as "the ability of a test to measure an underlying construct, that is the actual conceptual target of evaluation" and argues for a "benchmark-validation-first" culture in which a benchmark's validity is established against real-world data before it is used to judge model quality [27]. The paper's domain is medicine rather than coding agents, and the name is all it lends here; the cases above are the argument.

Two structural changes follow from those cases rather than from the vocabulary. The first is to report suites instead of single scores: benchmarks are valuable tools for understanding a model when they are not used for ranking, and curating them into benchmark suites gives practitioners the range of trade-offs that a single leaderboard number collapses [28]. The second is to decompose the score. If a number is produced by a model, a harness and an environment jointly, then a benchmark that cannot report the three separately is not reporting the model at all [1] — and the fix is a design change to the harness, not a fresh set of questions.

The independence half already exists institutionally, in pieces. METR reported GPT-5's 50%-time horizon as 2h17m with a 95% confidence interval running from 65m to 4h25m [29] — a capability number shipped with its own uncertainty, which is still rare on a leaderboard. The US AI Safety Institute signed formal memorandums of understanding for external testing with frontier labs in August 2024, and the EU AI Act's third-party conformity assessment for high-risk systems becomes fully applicable in August 2026 [29].

That independence comes with institutional limits. Third-party auditing today runs mostly on API access rather than weight access [29]. And when OpenAI describes its own external-testing programme, the contractual language reserves an assessor's research and publication rights only where such uses are "submitted to OpenAI for review and approval in writing prior to any publication" [30]. That is real independence, and it is independence exercised at the lab's discretion.

The SWE-bench audit numbers are one lab's account of one lab's benchmark, reported at second hand. The saturation result is a regression across 60 benchmarks, which is a statement about what predicts saturation and not about what causes it. Several of the harness and judge results rest on single papers or single vendor write-ups with no independent replication attached. Nobody has published the measurement that would settle the harness question outright — how much of any given leaderboard gap survives a change of scaffold, across models, with variance reported. That the most consequential quantity in agentic evaluation is unmeasured is itself the finding.

The field can keep buying difficulty by the year. Every purchase works, briefly, and expires on a schedule the last one has already demonstrated. Or it can start paying for validity — slower, unglamorous, and the only one of the two that compounds.