The New Unit of AI Capability Is the Configuration

What the claim is — and what it is not

Every deployed language-model agent runs inside a harness: the software layer that surrounds the model with system prompts, tools, context assembly, state, sandboxes, permission boundaries, retry logic, and verification loops. A recent academic survey formalizes this layer as a runtime design object in its own right — governing execution sandboxing, state checkpointing, verification, error recovery, and sub-agent coordination — and periodizes agent engineering into four paradigms, harness engineering among them [1]. Practitioners have converged on an operating-system metaphor: the model is the processor; the harness is the operating system that curates context, boots the session, and provides the drivers [2].

The claim of this essay is that this layer has become an independent capability layer: a place where measurable performance, cost, and risk are created and destroyed, with its own engineering discipline, its own measurement problems, and its own economics. That is a strong claim, so I will be precise about what it does not say.

It does not say that capability lives in the harness, or that the harness has displaced the model as the primary source of capability. The evidence reviewed below supports three narrower propositions. First, changing the harness while holding the model fixed can move measured performance by amounts that rival a model-generation upgrade. Second, the harness independently shapes cost, reliability, and governability, through mechanisms that do not run through model quality at all. Third, and consequently, the correct unit of evaluation for agentic systems is the model–harness configuration operating in a specified environment, not the model alone.

A boundary note, because the argument depends on it. The harness is not the environment: the sandbox, the repository state, the network access, and the task itself sit outside both model and harness, and results in this literature vary with all three. Nor is the budget an afterthought — token ceilings, time limits, and retry allowances change what any configuration can achieve, which is why the closing formulation of this essay attaches a disclosed budget to the unit of analysis rather than treating it as an implementation detail. When this essay says "configuration," it means the full tuple: model and version, harness and version, environment, and budget. One caveat on that tuple holds throughout: none of the evidence cited below varies the environment while holding the rest fixed — the environment axis is declared here, not demonstrated.

Throughout, I will grade evidence explicitly. Tier A evidence holds the model constant and varies the execution layer under controlled conditions — this is the only evidence that supports causal claims about the harness. Tier B evidence reports configuration-level results: the same model inside two different products, or a novel architecture achieving a strong result. It shows that configurations matter without isolating which component did the work. Tier C evidence is directional — engineering blogs, position and survey literature, practitioner commentary. The grade is two-dimensional: a tier reflects study design, and vendor authorship is flagged as an interest wherever it applies, whichever tier the design earns. Tier B and C evidence can suggest; only Tier A can begin to establish.

What the strongest evidence shows

The cleanest public result comes from LangChain's engineering team, published in February 2026. They took a fixed model — GPT-5.2-Codex — and changed nothing but the harness of their coding agent: system prompts that force build-verify loops, middleware that injects directory context proactively, loop detection to break repeated failing attempts. The agent's score on Terminal-Bench 2.0, an 89-task suite spanning machine learning, debugging, and biology, rose from 52.8% to 66.5% — a 13.7-point gain with no change to the weights, which moved the agent from outside the top 30 to the top five at the time of publication [3]. The percentages are the durable finding; the rank was a snapshot of a contested leaderboard and has since shifted.

Two details of that write-up deserve more attention than the headline number. The team found that agents have no natural tendency to enter a build-verify loop — verification had to be engineered in from outside. And when they varied only the model's reasoning-effort setting, running everything at the maximum "xhigh" mode scored 53.9% because of timeouts, uniform "high" scored 63.6%, and a staged allocation — deep reasoning for planning and verification, less for execution — reached the 66.5% final score [3]. The vendor's own maximum-effort dial, turned all the way up, was the worst of the three effort settings tested. Two caveats apply even here: this is a vendor reporting results on its own product, and it is one benchmark. But the design — model held constant, harness varied, scores reported per configuration — is exactly the comparison the causal question requires.

The second Tier A source is Harness-Bench (arXiv 2605.27922, May 2026), a factorial study from a Peking University and Qiyuan Tech team: 106 sandboxed tasks across eight workflow categories, run across six harnesses and eight model backends, producing 5,194 execution trajectories [4, 5]. Holding the task suite and model pool constant, swapping the harness alone moved aggregate scores by 23.8 points — the minimalist NanoBot configuration scored 76.2 against OpenClaw's 52.4, and NanoBot reached its score in a third of the turns that heavier stacks consumed; these figures are reported via Tier C commentary on the paper rather than verified against the paper's own text [5]. The paper's stated conclusion is the thesis of this essay in benchmark form: "agent capability should be reported at the model–harness configuration level rather than attributed to the base model alone" [4]. Its failure taxonomy points the same direction: the most common failure mode across trajectories was contract and format violations (36.4%), followed by tool errors without recovery (24.6%) — failures of execution alignment, where plausible reasoning decouples from tool feedback and verifiable output, not failures of reasoning per se [4].

A third line of evidence compares the two sources of variance directly. A position paper arguing for mandatory harness disclosure (arXiv 2605.23950) assembles the relevant observations: on SWE-bench Pro under a single standardized scaffold, six leading frontier models spanned only 4.9 percentage points (41.0% to 45.9%); holding one model fixed and varying the harness instead widened the gap to 9.5 points — roughly twice the entire between-model range [6]. The same paper collects cross-scaffold gaps from the Holistic Agent Leaderboard of 34 points or more for individual models on SWE-bench Verified Mini [6]. These are leaderboard observations assembled to support a position, not a randomized experiment, and the paper is explicit about that. But the structure of the comparison — model variance under a fixed harness versus harness variance under a fixed model — is the right structure, and in these samples the harness axis was wider. The same paper adds a small controlled factorial experiment across three models and proposes three measurable channels through which harnesses generate variance — stability of the execution loop, drift of the assembled context, and the lag between an anomaly appearing and the controller correcting it [6]. Whether or not those particular formalisms survive, they mark the shift that matters: harness effects are being decomposed into mechanisms and measured, rather than gestured at.

Finally, the model-fit result. LangChain now ships per-model "harness profiles" — the same scaffold retuned to each model family's conventions — and reports that on a subset of tau2-bench, a custom profile lifted GPT-5.3 Codex from 33% to 53% and Claude Opus 4.7 from 43% to 53% over the generic default [7]. Model held constant, harness varied, ten to twenty points of movement — with the now-familiar caveats: vendor-reported, benchmark subset, no external replication.

Taken together, the Tier A evidence supports a specific causal statement: on the benchmarks tested, execution-layer changes alone, with the model held constant, produced performance movements of ten to twenty-plus points — comparable to or larger than the spread between frontier models on the same tasks.

What the evidence does not yet show

The most widely circulated number in this literature is also the most misread, and correcting the reading matters more than the number.

In April 2026, Endor Labs' Agent Security League — built on the SusVibes framework from Carnegie Mellon, 200 tasks drawn from 108 real open-source Python projects — ran GPT-5.5 through two products in the same week. Through Cursor's agent, functional correctness was 87.2%; through OpenAI's Codex harness, 61.5% [8]. That is a 25.7-point configuration-level difference between two products using the same model — and that is all it is. Cursor and Codex differ simultaneously in tools, permissions, prompts, retry behavior, workspace handling, stopping conditions, and test integration; nothing in the setup isolates which differences carried the gap. Endor's own analysis attributes many Codex failures to specific limitations in how that harness handles certain repository structures and build systems, and notes a two-to-three-point non-determinism band on repeated runs [8]. The observation is genuinely important — the same model, deployed through two shipping products, delivered materially different capability — but it licenses a claim about configurations, not a measurement of the harness effect alone. The same benchmark found Claude Opus 4.7 scoring 87.2% through its own vendor's harness and 91.1% through Cursor's [9], which reinforces the pattern and inherits the same limitation.

The security dimension of the same data makes a subtler point: configurations do not just shift a score, they change the shape of the trade-off. The Codex configuration, while far behind on functional correctness, showed a much narrower gap between its functional and security scores than the leading Cursor and Claude Code configurations — Endor reads this as the Codex harness surfacing security-relevant reasoning more often, even as it completes fewer tasks [8]. Two configurations of the same model can occupy different points on a functionality-security frontier. A single headline number, from any configuration, compresses that away.

Three further boundaries on the evidence deserve explicit statement.

Vendor authorship. The cost study this essay relies on most — examined in the next section — is a vendor-authored controlled study of the Writer Agent Harness (arXiv 2607.06906): its authors are researchers at Writer, Inc., its corresponding author is Writer's CTO, and one of the two orchestration layers under comparison is Writer's own product [10, 11]. The LangChain results are a framework vendor's numbers on its own framework [3, 7]. None of this invalidates the findings; it changes their weight. A field whose strongest quantitative results come predominantly from parties selling the instrument under test is a field awaiting independent replication.

Sample sizes and external validity. The Writer study rests on 22 locked tasks [10]. Terminal-Bench 2.0 has 89 tasks; Harness-Bench's 106 tasks are sandboxed and offline by construction [3, 4]. Every headline number in this essay is an inside-the-benchmark result, and harness optimizations tuned against a suite can overfit it — LangChain's own hill-climbing write-up warns about exactly this reward-hacking risk [17].

Architecture results show possibility, not attribution. On ARC-AGI-1, an academic team wrapped DeepSeek V3.2 — open-weight, run in non-thinking mode, with no ARC-specific fine-tuning — in staged agentic harnesses: an Explorer-Definer Pipeline and a Reflective Orchestrator. The pipeline reached 57.50% pass@2 at $0.25 per task on the public 400-task set; the reflective orchestrator reached 67.25% at $0.62 [12, 13]. This is a striking configuration result in the domain where scaffolding was supposed to matter least, achieved at low reported per-task cost — although comparisons with the costs of test-time-compute-heavy approaches are not established in the cited record, and the result displaces nothing: it demonstrates an alternative regime, on one benchmark, with a design that does not isolate which architectural component earned which points.

One more discipline, applied throughout this essay: percentages are research findings; ranks are time-sensitive market information. A score of 66.5% on a versioned task suite is stable; "top five" was true in February 2026 and is not a property of the work.

Why the layer is economically durable

If the harness only moved benchmark scores, its importance would rise and fall with model quality. The stronger argument for treating it as a durable layer is economic, and the best-documented version comes from the vendor-authored Writer study introduced above — read with that provenance in view [10].

The study's design is clean: 22 locked enterprise tasks, six foundation models, two orchestration layers, everything else frozen. Swapping the conventional agent loop for the Writer Agent Harness cut blended cost per task 41% ($0.21 to $0.12), tokens per task 38% (14,200 to 8,800), and median wall-clock time 44% (48 to 27 seconds), while task-completion quality held at parity or slightly better (0.78 to 0.81) [10, 11]. Cost reductions ranged from 33% to 61% across all six models tested — the efficiency gains were model-invariant [10]. In the study's composite terms, quality per dollar rose 82%, and task completions per million tokens rose from 54.9 to 92.0 — the same models, doing more finished work per unit of spend, because the loop around them wasted less [10].

The mechanism matters more than the magnitudes, because the mechanism is checkable. The study names the disease "token maxing": buying capability with token growth — longer reasoning traces, more turns, wider tool payloads, larger replayed contexts — so that tokens per task grow faster than task value, a pattern masked by falling per-token prices [10]. Under a full-history replay model, where each turn resends the entire growing conversation, cumulative input tokens grow roughly quadratically with the number of turns; a harness that compacts context, maximizes cache hits, and offloads state converts that growth to approximately linear [10]. The authors present this as an idealized, approximate analysis, and it should be read that way — not every naive agent is strictly quadratic, and not every mature harness achieves linear. But the direction of the mechanism is arithmetic, not opinion, and it is amplified by provider pricing: cached prompt prefixes are served at roughly a tenth of the base input rate, so the harness that manages the cache largely sets the effective price [10].

The study also found that harness benefits are not uniform. Efficiency gains accrued to every model; quality gains correlated almost perfectly with the model's baseline strength (r = 0.99, across only six models — a directional finding, not a law) [10]. And harness features can fail model-specifically: sub-agent delegation reliability ranged from 0.86 and 0.85 for the strongest pairings down to 0.45 and 0.42 for others — below usable thresholds [11]. Both findings cut against harness triumphalism and for the configuration view: the same harness is a different system on a different model.

The second economic pillar is governance, and here precision matters because the temptation to overclaim is strong. Models can be trained toward economy and restraint, and increasingly are. What training cannot deliver is a guarantee. Hard per-task token budgets, generation fencing, permission boundaries, timeout enforcement, failure-spend ceilings, and audit retention are, by design, controls that bind precisely when the model's judgment fails — the Writer study, whose vendor sells the enforcement layer in question, is explicit that these guardrails must live in code, outside the model's judgment, because pathological loops corrupt exactly the judgment that would otherwise stop them [11]. Economic discipline can be encouraged through training, but hard budget, permission, and failure-spend guarantees must remain externally enforceable. That is a mechanism-scoped claim about where enforcement must sit, not a metaphysical claim about what weights can never learn — and it is the part of the harness layer that does not depreciate as models improve.

The absorption scenario, at full strength

The serious objection to everything above is absorption, and it deserves its strongest form.

Every published harness gain is a training roadmap. A scaffold that forces build-verify loops is a demonstration that verification behavior is learnable; an orchestrator that re-explores failed hypotheses is a specification for a reinforcement-learning reward. The labs are acting on this, deliberately. OpenAI's Codex models are trained inside a harness — shell-forward, with a deliberately limited tool set, learning to search, read, and edit through the shell during training itself [14]. NVIDIA's Nemotron 3 Ultra is post-trained through an explicitly agent-focused pipeline — supervised fine-tuning, reinforcement learning, and multi-teacher on-policy distillation aimed at long-horizon autonomous tasks — on an architecture chosen for agentic economics: a one-million-token context and up to roughly six-times-higher inference throughput than comparable open models [15]. On the research side, O-Researcher distills an entire multi-agent research scaffold — planning, tool calling, verification, peer review — into a single end-to-end model via multi-agent distillation and agentic RL, on the stated thesis that the open-closed gap is a data gap and scaffold trajectories are the data [16]. Philipp Schmid, whose harness-as-operating-system framing anchors the practitioner literature, draws the engineering conclusion himself: harness machinery should be built lightweight and modular precisely so it can be deleted as models improve [2]. Practice already reflects this churn: Manus rebuilt its harness five times in six months on the same models, and Vercel removed most of its agents' tools and got faster responses with fewer tokens for it [2]. Harness-Bench's own results rhyme with that — its top-scoring configuration was the minimalist one, completing tasks in a third of the turns of heavier stacks that scored lower [5]. The harness that wins is frequently the one that does less, and the one that does less is easier to absorb. On this view, any specific harness technique — the compaction heuristic, the verification prompt, the re-exploration policy — should be assumed absorbable within a model generation or two. If your moat is a trick, you have no moat.

That much of the absorption case I accept. Three things limit what follows from it.

First, absorption transfers behaviors, not fit. Among the models cited here, Codex is the most harness-absorbed, trained inside its own scaffold — and when Cursor deployed it, they had to re-engineer their harness around its training: renaming tools toward shell conventions like `rg`, preserving reasoning traces between calls (removing them measurably degraded Cursor's internal benchmark results), and adding explicit instructions to bias the model toward action [14]. Training behaviors into weights did not remove the need for a harness; it created a compatibility surface that any other harness must now match. LangChain's per-model profiles are the same fact from the framework side: as labs bake different conventions into different model families, the optimal scaffold becomes more model-specific, not less [7]. So far, absorption has differentiated the harness layer rather than shrinking it.

Second, the optimization loop stays outside the weights. LangChain's "Better-Harness" recipe uses eval suites as the signal against which prompt updates, tool additions, and structural edits are autonomously discovered and validated, with the same evals serving as regression tests [17]. This is an outer-loop, gradient-free optimization process over the scaffolding, with evals serving as the objective and regression test — not gradient descent, and the difference matters: it requires no access to weights, runs at deployment cadence rather than training cadence, and its artifacts (the eval suites, the discovered configurations) belong to whoever operates the loop. Absorption can eat the discovered configuration; it does not eat the discovery process.

Third, the governance floor does not absorb, for the mechanism-scoped reason given above: external enforcement is the requirement, whoever writes the enforced policy.

Now the honest concession. Tier C commentary on Harness-Bench reports lower cross-harness variance among stronger backends on its task suite — suggesting, but not establishing, that stronger models may be less sensitive to harness choice [5]. That observation traces to the commentary, not to the paper's archived conclusions, and the same commentary hardens it into an inverse law that the underlying data has not been shown to support [5]. But if the trend generalizes, the performance case for the harness weakens over successive model generations even as the economic and governance cases hold, and the layer's center of gravity shifts from capability creation toward cost and control. The Cursor-Codex evidence points the other way — absorption has so far increased model-harness coupling [14] — and the Writer study's finding that quality gains track model strength points both ways at once [10]. The compression scenario is live, and this essay does not claim to know which force wins.

Implications for procurement, standards, and competition

If capability, cost, and risk are configuration properties, then institutions that currently transact at the model level are transacting on incomplete information. The implications are concrete.

Procurement. An enterprise buying "GPT-5.5" is not buying a level of capability; the same model shipped 25.7 points apart in functional correctness across two products in the same week [8, 9]. A buyer who knows only the model name cannot distinguish those products, price that risk, or diagnose failures — when users of one coding product saw quality regress over a two-week window in the spring of 2026, the vendor's own investigation located the fault in the harness and agent-SDK layer, not the model [18]. Buyers should therefore require vendors to disclose, and version, the full configuration: the model and its version; the harness and its version; the tool set and its permission boundaries; the context and compaction policy; retry and stopping rules; token and time budgets; the verification method applied to outputs; and the evaluation environment behind any quoted benchmark. Disclosure must also be continuous, not one-time: harnesses ship updates on software cadence, weekly rather than yearly, so a configuration that was evaluated at contract signing may not be the configuration running a quarter later. Buyers should treat harness version changes the way they treat dependency updates — logged, notified, and re-testable against an agreed acceptance suite. None of this is exotic — it is the information the vendor's own engineers already use — and security practice is arriving at the same list from the other direction, locating agentic risk in the hooks, harnesses, and orchestration layers that connect agents to enterprise systems, where overly permissive tool access can render behavioral constraints meaningless [21].

Evaluation and standards. Leaderboards should report model–harness–environment–budget configurations, not model brands. This is now the stated position of the measurement literature: Harness-Bench concludes that capability should be reported at the configuration level [4], and the disclosure position paper argues for mandatory harness disclosure rather than forced uniformity — standardized scaffolds are one legitimate regime, but they measure a floor, and harness-side gains are real information that uniformity erases [6]. Princeton's Holistic Agent Leaderboard operationalizes part of this: standardized, cost-aware, third-party evaluation with logged traces — and its own results record a benchmark being effectively solved by a scaffold change, when an updated Claude Code scaffold with the same model family sharply outperformed the benchmark's reference agent [19]. Independent, configuration-explicit evaluation of this kind — like the Endor league's finding that functional correctness up to 84.4% coexists with secure-code generation peaking at 17.3% across the same league of configurations [20] — is what a buyer-side evidence base looks like. Much of the strongest evidence in this essay comes from evaluators rather than vendors.

Audit and oversight. For regulators and internal risk functions, the configuration view has one sharp consequence: controls that matter — spend ceilings, permission boundaries, stopping conditions, audit logs — should be verified at the harness layer, in code, rather than accepted as model properties or training claims [11]. An audit that certifies a model and ignores its execution layer certifies the smaller half of the system.

Competition. Two market structures are now plausibly in play. In one, model vendors integrate upward and ship the mode, not just the model — Codex trained inside its own harness is the clearest example [14], and Sam Altman has said, in remarks quoted in industry coverage, that he no longer thinks of the harness and the model as entirely separable things [9]. In the other, harness operators make models substitutable backends: at the time of the relevant report, one independent harness held three of the top ten Terminal-Bench 2.0 positions with three different vendors' models [18], and an independent harness outscored a model vendor's own official run of the same model by 17.4 points [18]. It is tempting to conclude from the configuration gaps that the harness operator now controls the customer's experienced capability and therefore the margin. The evidence does not support that inference — the gaps are benchmark-specific, the direction of integration is contested, and both structures are visibly being built at once. What the evidence does support is narrower and still consequential: harness differentiation may weaken model-level commoditization assumptions and create new switching costs at the execution layer. Which layer captures the value those switching costs protect is an open competitive question, not a settled one.

Trajectory data, under conditions

One asset class deserves separate treatment because it connects the harness layer to the next model generation: trajectories. The practitioner claim is that competitive advantage is shifting from prompts to the trajectories a harness captures — execution traces with tool calls, errors, recoveries, and outcomes, usable as training data [2]. The mechanism is real and demonstrated: O-Researcher's entire method is the conversion of multi-agent scaffold trajectories into training data for a single model, with staged filtering for trajectory quality [16]. Whoever operates the loop is in a position to collect the data that trains the successor — which is precisely why labs increasingly run loops of their own.

But position is not possession. The harness operator is often best positioned to capture trajectories — subject to data rights, privacy rules, retention policy, and sufficient deployment scale. Turning captured traces into a training asset additionally requires user consent and a lawful processing basis, preservation of the tool and verification signals that make a trace informative, enough task volume to matter, the ability to distinguish success from failure from lucky accident, and the contractual right to retrain on any of it. My searches for primary sources on how these conditions are being resolved in practice returned nothing citable, so I flag the gap rather than fill it: the trajectory thesis is mechanically sound and legally unpriced. It should be treated as a conditional advantage, not an automatic one.

The configuration as the unit of analysis

The findings of the past nine months do not show that capability has moved into the harness. They show something more precise and more disruptive to current practice: that capability, cost, and risk are jointly produced by a model, a harness, and an environment, and that attributing them to the model alone now misdescribes the systems being measured, bought, and governed. Model-held-constant experiments put execution-layer effects at ten to twenty-plus points on the benchmarks tested. A vendor's controlled study — read with its authorship in view — locates much of the cost structure of agency in the orchestration layer. Product-level comparisons show the same weights delivering materially different capability through different loops. And the deepest harness behaviors that labs train into weights have so far increased, not decreased, the specificity of the fit between model and scaffold.

The model is no longer a sufficient unit of analysis. For agentic systems, capability, cost, and risk must be attributed to a versioned model–harness configuration operating under a disclosed budget and verification regime. Benchmarks that report configurations, buyers that demand them, and audits that verify them are not bureaucratic overhead on the way to better models. They are the measurement discipline appropriate to what these systems have become.