Agent Harnesses Multiply What the Model Already Has — They Don't Supply It

Same weights, two scores

CORE-Bench asks an agent to do the thing every reviewer wishes someone had done first: take a published paper with its code and its data, run it, and reproduce the reported results. The Princeton benchmark ships 273 tasks drawn from 91 papers, sorted into three difficulty tiers [1].

Two agents were run against it on the same underlying model, gpt-4o. AutoGPT, a generalist, finished 26.7% of the tasks overall [1]. CORE-Agent — the same benchmark, the same model, an agent specialized to the task — finished 45.93% [1]. The gap is not confined to one tier. On retrieval tasks it is 35.6% against 57.78%; on the easy tier, 37.8% against an identical 57.78% — the repetition is the source's own table, not a typo; on the hard tier, 6.7% against 22.22% [1].

The weights were the same in both runs. What differed is the code sitting between the model and the work: how the task is decomposed, what the model is shown at each step, which commands it may run, what happens when a dependency install fails at minute forty. That layer has a name — the agent harness — and a definition considerably looser than the arguments built on it.

Results of this shape are now common enough to have produced a conclusion, and the conclusion is a slogan: the harness matters more than the model. It is on Reddit and on LinkedIn. It is on Medium, condensed to a headline: the model is the easy part. It is in a run of vendor rankings that need it to be true.

What is not there is a measurement. A well-measured phenomenon has acquired a badly-sourced conclusion, and the two have been travelling together long enough that the conclusion now gets cited as if it were the measurement. This piece takes the measurements seriously enough to disagree with the slogan they produced.

What a harness is, and what keeps getting billed as one

The most careful definition in circulation comes from a United Nations University report on engineering and governing the agent harness. It defines the harness as the runtime infrastructure surrounding one or more language models, doing four things: mediating between model output and external action, maintaining state across model invocations, imposing constraints on action selection, and externalizing memory, observation and recovery [2].

Three of those four are worth pausing on, because they name concrete objects. Maintaining state across model invocations matters because a model sees only its context window — the fixed span of tokens available in a single call — and long work does not fit inside one. Compaction, the practice of summarizing an exhausted context so the next call can continue, is the standard remedy. Constraining action selection means deciding which tool calls the model may issue, where a tool call is a structured request the model emits and the runtime executes; tools increasingly reach agents through the Model Context Protocol, which types those calls rather than leaving them to string parsing. Externalizing recovery means that when step nineteen fails, something other than the model decides whether to retry, roll back, or stop.

The same report draws a distinction that the slogan tends to collapse. There are three layers, not two: the environment is where the agent acts, the platform is the channel through which the agent is delivered, and the harness is how the agent is made able to act [2].

That separation earns its keep immediately, because the largest same-model swing anywhere in this evidence pool does not belong to the harness at all. A controlled experiment on agent-ready web design built two versions of the same e-commerce site — identical structure, products, pricing, inventory, ratings and checkout — and ran browser agents against both. Across GPT-4.1, Gemini 2.5 Flash and Grok 4 Fast, the agent-ready site produced 134 successful passes out of 150 against 74 for the baseline: 89.3% against 49.3%, a difference of 40 percentage points across 300 attempts [3].

Forty points, three models, one controlled change — and the change was the website. Credit that to the harness and the argument is lost before it starts; it belongs to the environment.

What the harness moves in every run is the bill

The cleanest attempt to isolate the harness's own contribution comes from Writer, Inc., evaluating its own harness in a paper on what they call the harness effect. Six foundation models — Claude Sonnet 4.6, Gemini 3.1, Gemini Flash 3.5, Qwen 3.6, GLM 5.1 and Palmyra X6 — were held fixed across 22 locked evaluation tasks, and only the orchestration layer was swapped: a conventional production agent loop out, the Writer Agent Harness in [4].

The result is lopsided. Blended cost per task fell from $0.21 to $0.12, a 41% cut [4]. Median wall-clock fell from 48 s to 27 s, down 44% [4]. Tokens per task fell from 14.2k to 8.8k, down 38% [4]. Quality per dollar rose 82% and task-completions per million tokens rose from 54.9 to 92.0 — and quality itself held at parity [4]. Their sharpest line is that the orchestration layer moved cost per task more than switching between the cheapest and the most expensive model did [4].

The harness that wins is the one whose authors built it, on tasks its authors chose. What survives that conflict is the mechanism, which is arithmetic rather than advocacy. Production agent loops report input-to-output token ratios near 100:1, which makes the input term very nearly the whole bill [4]. Providers serve tokens that repeat a previously seen prompt prefix from cache at roughly 0.1× the base input rate [4]. Prompt caching, in other words, is a tenfold discount available only on the portion of the prompt that has not changed since the last call — so the cache-hit fraction, the share of input tokens served from cache, is the single largest term an orchestration layer controls.

Nothing in that paragraph is something a model can do for itself from inside a forward pass. Whether the prefix is stable across calls is a property of how the runtime assembles the prompt. A model with no visibility into its own billing cannot choose to be cheaper.

So the first half of the answer, at the width the evidence actually supports. Swapping the harness moved the invoice hard, and moved it for every model in the set. In this particular comparison it also left the score where it was — but that is one comparison, against a baseline that was already a conventional production agent loop, on 22 tasks its authors locked. It is not a finding that a harness cannot move quality. The CORE-Bench pair above says it can, by nineteen points on the same weights, when the baseline agent is a generalist badly matched to the task. The difference between those two results is the subject of the next section: the cost effect was the same for every model, and the quality effect was not.

Where the quality actually comes from

The same study's efficiency gains were model-invariant: every one of the six models got between 33% and 61% cheaper [4]. The quality gains were not. They tracked baseline model strength, correlating with it at r = 0.99 across those six models — a pattern the authors name harness leverage [4]. The better the model, the more a good harness was worth on quality; the weaker the model, the less there was to release.

Six models is six models, and a correlation that near-perfect on that few points is suggestive rather than a law.

So the harness reads as a multiplier on the model rather than an addend: it does not add capability, it determines how much of the model's capability survives contact with a real task. Multiplying a small number still gives a small number, and the same multiplier is worth more the larger the number it acts on.

That reading needs corroboration the vendor paper cannot supply, and CORE-Bench supplies a partial one — not of the multiplicative form, which nothing measured here settles against its alternatives, but of the ceiling. Specialization more than tripled the hard tier, from 6.7% to 22.22%, and left the best agent on the hardest tasks at 21–22% [1, 6]. Both halves of that sentence are the finding. In relative terms the harness moved the hard tier further than it moved anything else on the benchmark; in absolute terms it barely moved it, and absolute completion is what decides whether the work can be handed over. On the hard tier there was less model capability for the harness to let through.

A second independent result cuts against the reflex that more harness is better. A paper defining marginal tool utility evaluated frontier models on 25 public observability tasks from APEX-SWE — a subset of 100 tasks, 75 of them private — across 150 agent trajectories with different tool suites [5]. The default harness, carrying three read-only tools, scored 0.32 [5]. Stripped to one tool, Grafana/Loki alone, it scored 0.36 [5]. Stripped of all its protocol tools, it dropped to 0.24 [5].

Removing two of three tools raised accuracy. On one narrow task family that is a direction rather than a rule, but within it the direction is unambiguous: harness contribution is not monotone in harness quantity. There is an optimum, not a slope. That constrains every account of harness contribution, this one included: whatever the harness is doing, it is not a quantity to be maximized.

Then why does everyone say the harness beats the model?

Because the phenomenon is real, visible and infuriating, and the people it happens to are not running controlled experiments.

The evidence base under the slogan is worth naming plainly. A Reddit thread reporting a nine-point swing on SWE-bench Pro, a benchmark of real GitHub issue-resolution tasks, with users in the replies reporting swings of ten to twenty points from scaffold changes alone. A LinkedIn essay whose headline swing of more than thirty points has no primary source, no named benchmark, and no method. A Medium post. Several vendor rankings whose business model is the conclusion.

A framework comparison reports a same-model swing on Princeton HAL's GAIA benchmark — GAIA being a general-assistant benchmark of multi-step, tool-using questions. Claude Opus 4 is reported at 64.9% inside one orchestration scaffold and 57.6% inside another, a seven-point gap from orchestration choice alone — a second-hand figure, reaching us through that comparison rather than from HAL directly [11].

Seven points is real. Seven points is worth engineering for; it is larger than the gap between many adjacent model releases. It does not amount to more than the model, and it is not the thirty-plus points the slogan advertises. The slogan is a correct observation about variance, promoted to a false claim about causation, and carried mostly by sources with no measurement behind them.

What happens when the model outgrows the scaffold

The real objection is the opposite of the slogan, and it is better measured than anything the slogan rests on.

A study in Nature Machine Intelligence ran 260 configurations across six benchmarks, five architectures and three LLM families, comparing multi-agent systems — several model instances coordinating, decomposing and cross-checking each other — against strong single-agent baselines [7]. Multi-agent systems did not reliably beat strong single agents. Performance varied heavily with problem structure and architecture. And the finding that matters here: the benefit of coordination diminishes as the underlying language model becomes more capable, with single-agent baseline performance emerging as a key predictor of when coordination helps at all [7]. The title states it without hedging: capable language models can outgrow the benefits of collaboration.

The general form of that observation reaches us second-hand, through a curated index of harness-engineering writing that distils an Anthropic essay on harness design for long-running application development to one line: every harness component assumes the model can't do something; those assumptions expire [12]. We have the index's summary, not the essay. Practitioners have drawn the obvious conclusion anyway, and they draw it as the bitter lesson replayed for agents: scaffolding you hand-build gets absorbed by the layer beneath you — a widely shared LinkedIn post puts it as half the agents shipped in 2025 being orchestration code the model no longer needs [14], a rhetorical accounting rather than a count of anything. The measurement under all of it is the Nature study.

The objection therefore lands on the thesis rather than beside it. If the multiplier is itself made of scaffolding, the multiplier has an expiry date, and the thesis describes a transitional period rather than a standing arrangement.

What gets absorbed, and what was never a deficit

An expiry date on what, though? The two measurements point different ways: Writer measured harness quality gains rising with model strength [4], Nature measured coordination-scaffold gains falling with model strength [7], and neither paper reconciles them. Our reading, which neither study states, is that harness names two kinds of work that expire on different schedules.

Compensatory harness work assumes a cognitive deficit and does the model's thinking on its behalf: decomposing a task the model could not decompose, running a second instance to check the first one's reasoning, coordinating specialists because no single call could hold the problem. That is precisely the class the Nature study measured, and precisely the class whose returns it found shrinking as models improve [7]. The objection is right about this half. A verification loop that exists because the model hallucinated file paths in 2024 is dead weight against a model that no longer does.

Non-cognitive harness work assumes nothing about the model's mind. It carries state across discrete context windows, keeps the prompt prefix stable enough to hit cache, types and validates tool calls, gates what the agent may touch, sandboxes what it runs, records what happened, and rolls the whole thing back when it goes wrong. Every one of the four functions in the UNU definition is of this kind [2]. These are properties of the boundary between a stateless text generator and a stateful world with money, credentials and side effects in it — facts about the deployment, not gaps in the model's reasoning.

The load-bearing witness on that point is a lab with maximal incentive to say otherwise. As quoted in the UNU report, Anthropic's framing emphasizes state handoff across discrete context windows and holds that even with frontier models, "compaction isn't sufficient" for production-quality work spanning multiple sessions [2]. We are reading Anthropic through the report's summary of it; taken at that remove, a model vendor is still not the party you would expect to argue that its model needs external machinery.

Read that way, "those assumptions expire" stops being the harness's obituary. It is a maintenance schedule: a standing instruction to re-audit which components are still compensating for something, which describes an engineering discipline with a lifecycle rather than one with an end date.

That discipline is already institutionalized, which is at least evidence that the people doing it do not expect it to evaporate. An exploratory study of harness engineering examined 2,853 repositories drawn from an initial 37,249 GitHub projects and found configuration surfaced as versioned, reviewed, repo-level artifacts: eight distinct mechanisms from static context files to dynamic subagents [8]. Context files such as CLAUDE.md and AGENTS.md were the most adopted mechanism, used by 61.5 to 100% of repositories depending on the tool; 493 repositories (17.3%) had standardized on the tool-agnostic AGENTS.md; 295 (10.3%) configured two tools and 50 (1.8%) configured three or more [8]. That is an adoption measurement and only that: it records what teams were doing in repositories active through 2026, and says nothing about which of those mechanisms will still be earning their place two model generations from now — adoption is not durability, and the study does not claim it is. What it does show is that teams are not currently treating the harness as scaffolding to be discarded. They are checking it into version control.

The constructive case closes the loop back to harness leverage. NVIDIA's NOOA framework reports 82.2% on SWE-bench Verified — the human-validated subset of the SWE-bench issue-resolution set — running GPT-5.5 through a general-purpose 253-line agent with no benchmark-specific prompts, above the 79.2% published leaderboard result at submission, and 79.8% with Opus 4.6 [9]. It reaches that number in 29 LLM calls and roughly 1.1M tokens per task, where comparison harnesses needed 66 calls and 2.2M tokens to reach 78.2% [9].

Better and cheaper, on a strong model, from a harness that is smaller rather than larger. NVIDIA is reporting on its own framework, but the direction is the one the cost and quality results predict.

So how much, then — and what to do with an undisclosed number

On the measurements available, the answer to "how much of an AI system's capability comes from the harness" is not one number, because the harness moves two different currencies at different rates.

It moves cost and tokens hard, and largely independent of which model is running underneath — 41% on blended cost, 38% on tokens, with every model in the tested set landing between 33% and 61% cheaper [4]. It moves quality in proportion to the model's own strength [4]. It can nearly double an agent's score on tractable tasks and still leave the hard ones near 22% [1]. The buyer's question was never which one matters more; it is how much of what they are paying for reaches the task.

Which makes the most portable consequence a reading rule rather than a purchasing rule. A benchmark number reported without its harness is not a claim about a model. It is a claim about one model inside one scaffold, and the same scaffold is not what the reader will be running. As one harness comparison puts it, "a model score is not a product score": a result obtained inside one scaffold has not thereby been obtained inside every compatible harness [13].

This is why the least glamorous part of benchmark design turns out to matter more than leaderboard position. CORE provides a harness that creates a virtual machine per agent-task pair with standardized hardware, runs the agent there, and downloads the results, so evaluations are parallelizable and isolated [1]. REAL Bench ships its evaluation harness through the AGI SDK and pins agents against a fixed mini-Internet of 11 replica sites and 112 standardized tasks [10]. Neither of those choices will ever be the headline. Both are what make the headline mean anything.

On the evidence assembled here, the harness does not so much supply capability as decide how much of it arrives — a reading rather than a measured law, but the one every result above rewards. An agent number quoted without the harness it ran in has quietly dropped the second half of that sentence.