The Description Is Always Loaded: How Agent Skills Break Tasks Without Being Invoked

A task the agent used to get right

An agent answers a hard question about a Treasury financial report correctly. Someone adds a folder of skills to its working directory, the kind of curated procedural guidance that is supposed to make it better at exactly this. It runs the same question again and gets it wrong. Nothing else changed, and on that run it never opened the skill.

The question came from OfficeQA-Pro, one of the two benchmarks in the study, which evaluates complex question answering over U.S. Treasury financial documents, and where the hard part is usually grounding — finding the right table, vintage, and definition — rather than the arithmetic [1]. An agent has to retrieve a figure from long enterprise PDFs, read text and tables, compare across years, and only then compute. The sum at the end is the easy part. Deciding which column of which report in which vintage is the answer is the task.

The broken run is not an anecdote about one unlucky task. It is a category the study is built to count. The work separates two outcomes against a no-skills baseline: a regression is a task solved without skills but failed after skills are added, and a residual failure is a task that fails both with and without them [1]. The first is a cost the aggregate score is very good at hiding, because a regression and a gain of equal size cancel out in a mean.

Each of three model-harness stacks runs 486 tasks, 94 from OfficeQA-Pro and 392 from SpreadsheetBench, under four conditions, giving 5,832 runs in total [1]. A harness here is the agent scaffolding around the model, the loop that decides what the model sees and which tools it can call, so a model-harness stack is one specific pairing of the two. These are paired comparisons on the same tasks, not thousands of independent trials: every task is run with no skills and again with each of three skill libraries, and the comparison is between those runs of that same task.

Two features of the design bound everything that follows. Each condition is run once per task, with no replication across seeds, so run-to-run variance is not estimated. And the mechanism labels applied later are an observational classification grounded in paired trajectories and cross-library contrasts, not a controlled causal test [1]. The counts below are directly observed. The explanations are read off traces.

Summed over all library conditions the study observes 553 gain transitions and 324 regression transitions, which means regressions offset 59% of gross gains, leaving a net of 229 [1]. These are task-condition transitions rather than unique tasks: the same task can contribute a transition once per library, so the totals count events, not distinct broken tasks. Across the eighteen conditions the regression count ranges from 2 to 41, and no condition comes in at zero [1]. Every library tested breaks something.

The consequence shows up most sharply in one stack. On Claude Code with sonnet-4.6 on OfficeQA-Pro, the three libraries gained 10, 11 and 12 tasks respectively, and regressed 2, 4 and 7 [1]. Rank those libraries by gains and one order comes out. Rank them by net and the order inverts: the library that gained most also broke most, and finished behind the one that gained least. The paper's own summary of the pattern is that the best performing skills outperform others primarily by regressing less, not by gaining more [1].

So the question is not whether skills help. It is what, exactly, is doing the breaking.

The description is always in the room

The answer is structural, and it is visible in the file format before it is visible in any measurement.

A skill is a directory containing a SKILL.md file, which must begin with YAML frontmatter carrying two required fields, name and description, followed by a markdown body of instructions, and optionally bundled reference files and executable scripts [2]. That is the whole format. Its behaviour is governed by a three-level progressive disclosure architecture: metadata is always loaded, instructions are loaded only when the skill is triggered, and additional resources and code are loaded as needed via bash commands [3].

Level one is the frontmatter. Those two fields are what the agent sees unconditionally, because level one metadata is always loaded, present from the start of the run rather than fetched when the agent decides the skill is relevant [3]. In Microsoft's independent implementation of the same open format, the equivalent stage is named Advertise and budgeted at roughly 100 tokens per skill, injected into the system prompt at the start of each run so the agent knows what is available [4]. Level two is the body. When a skill is triggered, Claude uses bash to read SKILL.md from the filesystem, bringing its instructions into the context window [3]. Level three is everything bundled alongside, read only when the body points at it.

A skill is two objects, not one: a permanently mounted advertisement and an on-demand manual.

The advertisement is unconditional. It is in context because the skill exists, not because the agent decided the skill was relevant. The manual is conditional, and the whole architecture exists to keep it that way. Progressive disclosure is the core design principle that makes Agent Skills flexible and scalable [2], and it is a good design principle: an agent with a filesystem does not need to hold a manual in context to be able to open one, which is what makes the amount of material a skill can carry effectively unbounded. The cost is a side effect of that decision, not a bug someone shipped carelessly. To keep the manual out of context until it is needed, something has to stay in context permanently to announce it. That something is a sentence describing what the skill does and when to use it, written to be persuasive to a model deciding what to do next.

Which means a skill can act on a task it was never selected for.

Presence, not invocation

The study has a name for it. Osmosis is a skill changing behaviour without being invoked, and methods that act only on retrieval or invocation cannot see this channel [1].

The measurements that make osmosis more than a hypothesis are about engagement, and the striking thing about them is what they do not depend on. Whether a skill body gets opened at all turns on the harness and the benchmark, not on what the skill says. On SpreadsheetBench, OpenCode with minimax-m2.7 invoked a skill on only 5 to 14% of tasks, while Codex with gpt-5.4-mini read a skill file on 0 to 1% of SpreadsheetBench tasks [1]. The same libraries, on the other benchmark, are engaged on the large majority of tasks by those same stacks. In the paper's phrasing, the same library can be engaged almost everywhere in one setting and almost nowhere in another.

Now hold those two facts together. On one benchmark a skill library is opened on approximately none of the tasks. On that same benchmark, the same library still moves the score, in both directions. Whatever is producing that movement cannot be the manual, because on those runs nobody read the manual. It is the advertisement.

There is a well-evidenced alternative story, and it deserves the pushback it implies: skills go wrong because the agent picks the wrong one, and the answer is better selection. That story is real and it is measured carefully. Performance degrades as libraries grow — by up to 21% when scaling from a small set of known-helpful skills to a 202-skill library — and that drop decomposes into two parts: skill shadowing, where distractor skills displace the oracle skills at selection time, and context overhead, where the enlarged context degrades execution even when selection is correct [5]. The asymmetry is the finding: shadowing grows with library size and dominates, while context overhead stays small and is indistinguishable from zero, across a dataset of 2,545 trajectories [5]. Selection failure, not context bulk, is the primary bottleneck there.

A second study walks the same problem down a ladder of increasingly realistic settings, and every rung is a selection or retrieval problem. Force-loading curated skills yields 55.4%; letting the agent decide which to load drops it to 51.2%, with the same skills available; adding distractors takes it to 43.5%; making the agent retrieve from a pool containing the curated skills takes it to 40.1%; and removing the curated skills from that pool leaves 38.4%, only 3.0 points above the no-skill baseline [6].

Both results are solid, and both describe problems that better retrieval genuinely fixes. That is exactly why osmosis matters. It is the part that survives the fix. Route perfectly, retrieve perfectly, invoke only when appropriate, and the descriptions of every skill you did not invoke are still in the context window, still describing procedures, still shaping what the model thinks this kind of task calls for. The adjacent literature is measuring the visible half, and measuring it well.

Presence alone changes behaviour.

What the procedure displaces

Influence without invocation tells you which tasks a skill can reach. It does not tell you what the skill does to them. For that, the study classifies its regressions, and names two mechanisms alongside osmosis: grounding displacement, where a skill's prescribed procedure overrides how the agent interprets its inputs, and verification displacement, where the procedure suppresses checks the agent would otherwise perform on its outputs [12].

Put those next to the benchmark and the shape of the damage becomes obvious. On OfficeQA-Pro, the hard part is usually grounding — finding the right table, vintage, and definition — not the arithmetic [1]. A skill written to help with such a task will, in practice, tell the agent how to compute: normalise the units this way, apply the ratio in this order, format the answer like this. That guidance is not wrong. It is simply spent on the stage that was never the bottleneck, and it arrives at the expense of the stage that was. The agent that would have paused over which of three similarly-named tables is the right vintage now has a procedure to execute, and executes it, on the wrong table.

Verification displacement is the same move applied one stage later. An agent left to its own devices will often re-read its answer against the source, sanity-check a magnitude, or notice that a ratio came out impossible. A procedure that says what to do rarely says what to check, and a context that is full of what-to-do crowds out the habit of checking.

A skill spends the agent's attention, and existing skills spend it on procedure.

The full scheme is applied to all 81 OfficeQA-Pro regressions, where answers are single numbers graded against ground truth with a numeric tolerance, so a regression is a real failure and the trajectory usually shows why; SpreadsheetBench gets a coarser breakdown, because many of its regressions are a value-versus-formula grader artifact and cell answers do not support the same input-versus-output coding that separates grounding from verification [1]. And the labels themselves remain an observational classification grounded in paired trajectories and cross-library contrasts rather than a controlled causal test [1]. What is measured is that the tasks broke. What is inferred is where in the pipeline they broke.

Everyone's average has the same shape underneath

None of this is one paper's story, and the useful way to read the other work is not by result but by how much of the two-sidedness each study was in a position to see.

SkillsBench is the largest and the friendliest to skills. In its 84-task release, curated skills raise the average pass rate by 16.2 percentage points [7]. That is the number that travelled. The same work reports, in the same abstract, that 16 of 84 tasks show negative deltas, the worst being taxonomy-tree-merge at 39.3pp below baseline and energy-ac-optimal-power-flow at 14.3pp below [7]. The mean and the casualties were published together. Almost every downstream write-up carries only the mean.

SkillLens, from Microsoft, was designed to look at the distribution rather than through it. Across five domains, six target models and five extractors, model-generated skills help in 75% of extractor-target pairs and produce negative transfer in 25%, with ALFWorld the most fragile domain at 47% [8]. Negative transfer here means the skill made the receiving model worse than no skill at all. The same skill, in the same domain, gives GPT-5.4 a transfer effect of +4.93 and drops Qwen-9B to −1.69 [8]. Their own summary of it is that the same skill, different consumers, produce opposite outcomes.

The third study explains the shape of that variation rather than adding to it. On 15 MCP-grounded offensive-security CTF challenges across 180 trajectories, the comprehensive-skills condition consumed roughly 61 times more procedural-context tokens than the no-skills condition, 36,001 against 591, for a non-significant gain of 8.9pp with a 95% confidence interval running from −6.8 to +24.6pp [9]. The authors' reading is that the marginal benefit of curated skills is inversely related to the bandwidth of deterministic environment feedback [9]. Where the environment tells the agent loudly and immediately whether it is right, procedural guidance has little left to add. That also explains SkillsBench's spread: the same benchmark that gives healthcare 51.9pp gives software engineering 4.5pp [9], and software engineering is the domain where a compiler answers back.

The point is not that many papers agree. It is that every one of them reports a mean with a two-sided distribution underneath it, and only the studies that went looking for the negative tail found it.

The number that hides the number it needs

Why does the standard report not surface any of this? The answer is arithmetic.

For a fixed task set, the pass-rate difference is exactly gains minus regressions divided by the number of tasks. The aggregate therefore already reports the net effect. The only thing it discards is the split. Which is precisely why two libraries with the same average gain can still differ in how many tasks they break, and nothing in the average distinguishes them [1].

The study's own statistics make the case without any appeal to good practice. Because the treatments are paired against the no-skills condition, the net effect is tested with an exact McNemar test, which runs on the discordant pairs: the tasks that flipped, in either direction. Those are exactly the gains and the regressions. Tasks that pass in both conditions or fail in both drop out of the test entirely. The statistic that establishes whether a library helped at all is computed from the decomposition the headline number throws away.

Five of the eighteen conditions reach nominal p < .05: Claude Code with sonnet-4.6 on SpreadsheetBench under all three libraries, at +43, +43 and +47 with p < .001; the same stack's anthropic library on OfficeQA-Pro at +8, p = .039; and OpenCode with minimax-m2.7's anthropic library on SpreadsheetBench at +27, p = .008 [1]. The other thirteen confidence intervals include zero. And with eighteen simultaneous comparisons, a Bonferroni correction divides the conventional threshold by eighteen, which almost nothing here clears.

So the decomposition is the robust part of this work and the per-condition significance is not, which is an argument for reporting both rather than for reporting the mean alone. That is also where the broader measurement literature lands. The standard advice for evals is to treat them as experiments: compute standard errors, use clustered standard errors when questions are drawn in related groups, and size the eval to the effect you want to detect, with at least 1,000 questions recommended for good signalling ability [10]. An eval of 94 tasks run once per condition is not going to produce tight intervals, and reporting a single mean from it produces something worse than a wide interval. It produces a number with no visible width at all.

The split is not extra detail attached to the measurement. It is the measurement.

Write the checking, not the recipe

If procedure is what skills supply, and procedure is what displaces the stages that decide the answer, what should a skill contain instead?

The study's own answer is the mirror of its diagnosis. Existing skills overemphasise procedural guidance, the stage least often responsible for failure, while under-supporting grounding and verification, which are the dominant sources of remaining errors [12]. After correcting evaluation artifacts and studying traces, the authors find many regressions and persistent failures recoverable through better grounding and verification [12]. Not through better procedures. Through better instructions about what to read and what to check.

What carries the utility is content, not shape. Rewriting the same skill into different surface formats yields statistically indistinguishable downstream gains, with a paired test at p > 0.34, while each of three validated meta-skill dimensions — failure-mechanism encoding, actionable specificity, and a high-risk action blacklist — individually raises the better-rate above 64% [8]. Every one of those dimensions is about failure and constraint. None is about steps.

Which is why looking good is not a test. A skill that reads well is not thereby a skill that works, and the obvious automated shortcut for telling the difference is anti-correlated with the truth: an LLM judge asked to compare two skills picks the better one only 46.4% of the time, worse than chance, and its accuracy drops as the actual utility gap grows [8].

If a skill cannot be graded by reading it, the remaining discipline is subtraction, and subtraction has its own evidence. Tasks given 2 to 3 skills show the largest improvement at 18.6pp, while 4 or more skills provide only 5.9pp, and comprehensive skills actively hurt at 2.9pp below baseline [7]. If the advertisement is the channel, then every skill you install is a permanent claim on the context, whether or not you ever use it.

Nor is the shortfall one that authorship can automate away, because models are better consumers of procedural knowledge than authors of it: when prompted to generate their own procedural knowledge before solving tasks, models come in at 1.3pp below the no-skills baseline [7]. The automated skill-generation pipelines that are being evaluated by average gain are producing the artifacts this whole piece is about.

The stakes are larger than one paper, because this is the layer the field has committed to. The reviewed progression runs from weights to context to harness, with skills as the mechanism by which procedural expertise is externalised out of the model and into the runtime [11]. That bet may well be correct; the case that agent progress now depends on external cognitive infrastructure as much as on stronger models is a serious one [11]. But the instrumentation for the layer still reports one number per library, and one number is exactly the quantity that cannot see what the layer costs. A library that gains 12 and breaks 7 and a library that gains 10 and breaks 2 are not close calls that a bigger sample would resolve. They are different products, and the standard report renders them nearly identical.

Write the checking, not the recipe. And count both columns.