The Cache Miss After the Gap

The average is not describing the call that misses

A coding agent re-sends almost everything it has already said. Every step replays the conversation so far — the files it read, the tests it ran, the reasoning it produced — and the serving stack underneath is built so that the replay is nearly free: it keeps the model's intermediate state for tokens it has already processed, the KV cache, and reuses it instead of recomputing. By the headline measure, that works.

"In aggregate, prefix caching looks great: 95.7% of input tokens hit the cache." [2]

— Kan Zhu and colleagues, TraceLab

The number comes from a single trace — among the sources reviewed here, the only large cross-provider record of real coding-agent traffic anyone has released. Between September 2025 and June 2026, a group at the University of Washington instrumented its own daily use of Claude Code and Codex and published the log [1]. It runs to roughly 4,300 sessions and about 350,000 LLM steps [1]. Those sessions came from 43 developers [1]. They carried 430,000 tool calls in all, across both agents and more than 20 model versions [1]. That breadth is the point: this is not one vendor's telemetry about its own product. The authors say plainly that every session comes from one lab, and that limit rides along with every trace figure below.

That 95.7% average, the same authors go on, hides a sharp split between the two ways a step can start [2]. A step that begins with a tool result — the agent ran a test, read a file, got output back, and is continuing — hits at about 97.5%. A step that begins with a fresh human message hits at about 84.4%, because prompt caches expire after a few minutes of inactivity [2]. The cache does not know the difference between the two; the clock does. Those two classes are not equally common: 88% of all LLM rounds in the trace respond to a tool result rather than to a person [2]. What separates the two is not the model or the prompt. It is whether anything happened in between.

A second trace, collected independently and reported by the vLLM project with Mooncake, has the same shape. It covers 610 Codex traces on SWE-bench Pro [4]. The observed cache hit rate there is 94.2% [4], and the median trace runs 33 turns at an input-to-output token ratio of 131:1 [4]. That ratio is the workload in one number: an agent is overwhelmingly a reader. Each turn adds an average of roughly 2,242 tokens [4]. By turn 30 the context has reached about 80K tokens [4]. At the extreme it passes 180K [4]. The context only ever grows, and it grows in small increments. An agent says a great deal and very little of it is new. Those figures are vendor-measured on the vendor's own integration, so they describe the workload rather than settling anything about a system. Two records, two vendors, the same lopsided shape.

Put the two together and the puzzle is not "why is the hit rate low." It is why, at rates that high, several vendors and research groups spent 2026 rebuilding their serving stacks around the cache at all.

What a call-weighted mean hides

The useful decomposition is the one the trace performs: every LLM call in an agent session is either a later-call — a warm continuation that arrives moments after the previous one, over a prefix the server was almost certainly still holding — or a first-call, the step that crosses a gap. A first-call is either genuinely cold or, more often, a resume: state that existed, aged out, and has to be rebuilt.

A cumulative hit rate is a mean weighted by call volume, and the population is dominated by the class that could hardly miss. When 88% of rounds are tool-result continuations [2], moving the aggregate is mostly a matter of not breaking the calls that were already free. The class that misses is small enough to be rounding error in the headline metric and large enough to carry the bill.

The trace puts a price on it. Under an upper-bound counterfactual in which user-initiated steps keep their prefix cache instead of losing it to the human gap, cost falls by $5,189 [3]. That saving is 12.8% of final cost over priced rounds [3]. It is an upper bound, and it comes from the same one lab's sessions.

The same asymmetry shows up in what a step actually carries. Of the 2.34 billion append tokens in the trace, only 19% are genuinely new [2].

"Every new token is prefilled roughly 5× on average." [2]

— Kan Zhu and colleagues, TraceLab

Prefill is the read side of a call: the model processing the context before it writes anything. Stated as a ratio, that repetition is a prefill amplification, and the trace reports it per agent alongside what a median step reads and writes.

A median step replays roughly two orders of magnitude more context than new context [1].

The floor is the column worth staring at. It is what the same workload would cost if nothing were ever evicted, and neither agent is anywhere near it.

One caution before these numbers travel any further. They are measured in different ways. The 95.7% is a token share from production billing fields [2]. The 94.2% is a per-trace rate on a benchmark-derived workload [4]. Several figures below come from simulators and idealized models. They agree about shape. They are not points on a common axis, and this piece does not treat them as one.

Two clocks, three orders of magnitude apart

If the misses concentrate on the step that crosses a gap, the question becomes what the gaps are made of. The trace answers with two distributions that have almost nothing to do with each other.

The first is the tool clock. Tool-call durations in coding agents span several orders of magnitude, from hundreds of milliseconds for a file read to minutes for human input or a subagent invocation, and the variance exists within a tool type as much as across types — the same shell command may be a sub-second `git checkout` or a multi-minute compile [5]. The distribution is brutally top-heavy. Calls under a second make up about 61% of all calls [2]. Between them they account for only about 1% of total tool time [2]. The calls lasting over a minute are roughly 4% of the population [2]. They consume about 85% of the clock [2]. Almost all the waiting is done by a handful of calls. AMD, describing the workload its own stack was rebuilt for, notes that tool gaps in agentic turns are often close to a second, during which the accelerator sits idle [7]. So the tool clock is not really one clock either. It is a mass of gaps too short to react to, plus a handful long enough to change what the server ought to do.

During that gap the agent's KV cache is not garbage. It is the exact prefix the next inference needs, sitting in GPU memory doing nothing — the intervening tool execution, as MORI puts it, "is part of the same logical execution: the agent is waiting for feedback that will be embedded in the next prompt, often over the same accumulated context" [5].

The second is the human clock, and it is three orders of magnitude longer. Per session, total elapsed time averages 8.2 hours [1]. Human thinking accounts for 7.6 of them [1]. A coding session is mostly a person thinking. Most gaps between one request and the next are short; the ones that are not are very long.

The two clocks sit three orders of magnitude apart, and every interval on them is heavy-tailed.

Read the median column and the workload looks brisk. Read the column beside it and the same workload has to be held open for the better part of an hour.

A single LRU or TTL timer has to straddle both, and the policy that serves one clock is wrong for the other. KVFlow's designers describe the same mismatch from the scheduling side.

"Existing LLM serving systems typically adopt an LRU eviction policy, which becomes suboptimal under agentic workflows. Specifically, an agent that is about to execute may have been idle for a long time, while an agent that has just completed its execution might not be needed again in the near future." [9]

— KVFlow, NeurIPS 2025

The cost of getting this wrong was quantified before agents were the reason. InferCept, published at ICML 2024, measured what happens when a serving system treats every external interaction as the end of a request and forms a new one when the tool returns [8]. Recomputing already-computed context accounts for 37–40% of total model forwarding time [8]. Both available answers turned out to be expensive. On the same measurements, 37% of end-to-end execution time goes to recomputing KVs [8]. Another 27% of GPU memory is wasted [8]. The obvious alternative, swapping, wastes 26% of GPU resources [8]. Over 25% of total workload time is spent waiting for the swap [8]. Recompute and swap were not two strategies. They were two ways of paying for the same lapse. Handling interception properly was worth 1.6–2× serving throughput [8]. Neither of those measurements is about agents. Both are about what a serving system does when a request stops for a moment. That is the shape of the whole problem, measured two years before the trace above described the workload that makes it acute.

Retention has an exchange rate, and it is paid in memory

The obvious fix is to hold the state longer. The trace makes the price legible by sweeping the eviction timeout and reporting what each setting buys and what it holds.

Retention buys hit rate at an exchange rate of roughly seven times more suspended KV [3].

The middle row is where the sweep goes quiet, and it is the row the rest of this section is about.

Two conditions come attached. The sweep is an idealized model: a step counts as a total miss if its preceding idle gap exceeds the timeout and a full hit otherwise, driven by the observed human-think, tool-latency and generation distributions, with R derived analytically. No vendor's cache was measured. And it describes what retention would buy, not what retention costs in a system that also has to schedule around it.

Even so, two numbers line up in a way that is hard to ignore — though the link between them is ours, not any source's. The retention window an independent trace study finds cheap is the window Anthropic shipped as its default, and the step past that window is the one that carries a surcharge.

"By default, the cache has a 5-minute lifetime. The cache is refreshed for no additional cost each time the cached content is used." [10]

— Claude Platform documentation

A one-hour TTL is available at 2× the base input token price [10].

The knob is not uniform across platforms, and treating any one setting as universal is a mistake.

GPT-5.6's breakpoints are the newest of these, and the documentation describes them as especially valuable for agentic workflows [11]. Across the table, a read costs a tenth of base input — the discount most agent cost models assume, confirmed in the vendor's own price sheet rather than relayed through a blog. A price ratio is not a cost ratio; it embeds margin and demand. What survives is narrower and still sharp: the paid upgrade is exactly the regime the trace shows costs several times more resident KV.

A fixed timer is also the wrong shape for the traffic. SAECache reports that fixed-parameter eviction policies can degrade by up to 2.7× under workload mismatch [12]. Its adaptive alternative avoids that failure mode, and delivers a 1.4–2.7× improvement in time-to-first-token — TTFT, the delay before a caller sees anything — over production-style baselines [12]. Continuum takes the same view from the scheduler side, setting a per-request time-to-live from the reload cost and the queueing delay that eviction would induce, and arguing that the policy has to stay robust given how poorly tool durations can be predicted [13]. The disagreement is not about whether to hold state. It is about who is allowed to decide when it goes.

The open engines are growing the same vocabulary. Dynamo's KV Block Manager tiers GPU to CPU to disk, and — a small detail that says a lot about what tiering actually costs — only writes a block from CPU to disk if it has been touched at least twice, to spare SSD lifespan [14]. A vLLM RFC proposes a context-aware retention API in which the orchestrator annotates token ranges with eviction priorities and TTLs — "hard pinning doesn't scale," its author writes, and "what's needed is a way for the orchestrator to express which blocks matter more when eviction is unavoidable" [15]. It is a proposal on an issue tracker, not a shipped feature.

The state survived, and it is on the wrong worker

Paying for retention buys nothing if the surviving prefix is somewhere the next call cannot reach. Once a deployment is more than one instance wide, cache residency and cache reachability come apart, and the second one is a routing property.

"Without cache-aware routing, turn 2 of a conversation has a ~1/N chance of landing on the same worker as turn 1." [16]

— NVIDIA, on Dynamo's router

The measured version of that claim comes from llm-d, on a benchmark built squarely in the regime where placement decides everything: a multi-pod vLLM cluster whose total KV demand exceeded any single pod's capacity several times over [17]. Three published results point the same way, and in each of them the interesting column is not the gain.

Read the baseline before quoting the jump.

The first row is a vendor measuring its own scheduler on a synthetic workload it designed, so the honest reading is that it bounds how large the routing term can get, not how large it usually is. The third holds the most quotable placement number in this literature, and the easiest to misuse. Its baseline runs the read and write halves of a call on separate machines, across 12 GB200 GPUs, and the decode instance's KV was invisible to prefill, so only the system prompt was ever cached [4]. What that pair measures is a placement mistake.

The more informative half of the same work is the scale test, where a distributed KV pool held above 95% hit rate from 12 to 60 GPUs even under round-robin routing [4]. Make the state reachable and routing stops being load-bearing. "If we can cache those prefixes," the authors write, "prefill for the cached portion becomes essentially free. The true per-turn cost is only the new delta" [4]. Reachability, not capacity, is the variable that experiment moved.

Placement is now a first-class modeling axis. AgentServeSim, a simulator that reproduces real-system behaviour within 6% error on commodity CPUs, is built around exactly two things: a session-aware router that maintains program-to-instance affinity, and a KV residency model that tracks policy-defined placement across HBM, host DRAM or CXL, and eviction [19]. Claims about what any of this beats need their baseline's date attached. Automatic prefix caching is standard in current vLLM, hash-based, matching on full blocks only, with sha256 as the default hash since v0.11 [20]. A result measured against last year's default is a claim about last year.

Whether placement or scheduling is the larger lever is the question this literature cannot yet settle. Routing results and scheduling results are measured against different baselines, by different groups, in different topologies, and among the sources reviewed here there is no real-system ablation of agent-aware scheduling layered on top of good prefix caching at scale. The claim that survives all of them is narrower than any single headline: in every topology measured here, the variable that set the hit rate lived outside the attention kernel.

One geometry breaks the assumption entirely. In multi-LoRA serving, unique adapter activations cause KV divergence across agents, so a shared context does not produce a shared cache and ordinary prefix caching stops working; ForkKV's answer is to fork the cache with copy-on-write semantics, splitting a massive shared component from lightweight per-agent components, for up to 3.0× the throughput of state-of-the-art multi-LoRA systems [21]. It is a design response to a structural problem, not a measurement of production traffic.

Down a tier, and what the tier costs

Everything above spends memory. What makes it affordable is that there is somewhere cheaper to put the state, and by 2026 the four-tier hierarchy is the assumed substrate. AMD's UMBP manages a single logical cache spanning engine HBM, host DRAM, a shared DRAM pool and SSD, and plugs into SGLang as a HiCache L3 storage backend plus a KV-event feed to the scheduler [7]. SGLang's HiCache itself extends RadixAttention with three tiers — GPU, host memory, distributed storage — organised by a HiRadixTree [22]. There, a `--hicache-size 30` allocates 30 GB per rank [22]. Across eight ranks that is 240 GB [22]. Those are configuration figures rather than results, and the point is only that the knob exists. Dynamo's KVBM covers the same GPU-to-CPU-to-disk path [14]. Different vendors, different names, one shape: hot state on the GPU, colder state falling downward through cheaper memory.

The tier below is bigger, but not by as much as the phrase suggests. A typical H100 server pairs 80 GB of HBM per GPU across eight GPUs with about 1 TB of host DRAM [5]. That is roughly a 2× capacity gap between the tiers, where "offload to host memory" tends to imply an order of magnitude [5]. The hierarchy buys room. It does not buy a different order of magnitude of room.

The exchange rate also has preconditions, and they are the part that gets dropped when the headline travels.

The first is that the restore has to hide behind work the GPU was doing anyway. ScoutAttention's framing of prior offloading work is that it either demands frequent GPU-CPU transfers or imposes heavy CPU computation, and either way GPU utilization suffers while the system waits on I/O [24]. Its own answer, pre-computing one layer ahead on the CPU, is worth a 2.1× speedup over existing offloading methods while staying within 2.4% of baseline accuracy [24]. The trick there is not the transfer. It is the timing. TTKV makes the same argument about tier interaction, using block-wise streaming attention to overlap communication with computation [25]. On 128K-context tasks it reports 5.94× less cross-tier traffic, with up to 76% latency reduction and 2× throughput [25]. That is long-context tiering rather than agent serving, so the mechanism transfers and the workload does not. Both papers are saying the same thing in different vocabularies. A tier only pays if the transfer happens while the GPU is busy with something else. TokenCake states the condition as an explicit test: offloading a stalled agent's cache pays only if the predicted stall is long enough to cover the round trip, if there are waiting requests that can use the freed GPU blocks, and if the later upload will not crowd out more important work [6]. That is a test, not a policy, and its answer changes with the traffic.

"The Temporal Scheduler converts this idle period into a productive scheduling window by offloading the stalled agent's KV Cache to CPU memory and uploading it back before the agent resumes, avoiding the costly recomputation that a naive eviction would require." [6]

— TokenCake

The distinction that separates a working tier from a slow one is prefetch versus reaction. KVFlow's whole contribution is anticipating which agent runs next and loading its tensors in background threads before it is invoked [9]. Measured against SGLang with a hierarchical radix cache — that is, against a system that already had the extra tier — it is worth up to 1.83× for single workflows with large prompts, and 2.19× with many concurrent workflows [9]. A tier that is loaded reactively, on the miss, buys capacity and pays for it in stalls.

The second precondition is headroom, and it fails abruptly rather than gracefully. DUAL-BLADE's account of file-based NVMe offloading is that it leans on the kernel page cache, which produces cache thrashing, unpredictable latency and high software overhead under memory pressure [26]. Its NVMe-direct path bypasses the filesystem by mapping KV tensors to contiguous logical block addresses, cutting prefill and decode latency by up to 33.1% and 42.4% [26]. SSD utilization improves 2.2× [26]. The failure it fixes is not slowness but unpredictability. That is measured on edge-class hardware under tight memory budgets, which is where the cliff lives — but it is the same page cache underneath a datacenter file store.

Which number you get from the extra tier depends on where you are standing. Three groups have published on the same component, and no two of them measured the same thing.

Nobody disagrees about HiCache; they report different metrics from different operating points.

All three are vendor or partner measurements of the same component. KVFlow's 1.83× is measured against that same component as its baseline [9]. Both readings are true of different operating points. The variable is concurrency, prefix length, and whether the tier is prefetched or reactive. The four-tier hierarchy is a set of thresholds, not a capacity multiplier.

The latency target assumes a human is watching

The tier below HBM only pays if the restore hides inside a model forward pass. Chunked prefill competes for that same forward pass: both are decisions about what the GPU does while an agent is not producing tokens for anyone. That joint framing is ours, not any source's. What one source does measure is that the second of those policies is calibrated for a reader who is not there.

Every major serving system enforces time-to-first-token and time-per-output-token targets, and the mechanisms that satisfy them — chunked prefill, prefill-decode disaggregation, latency-capped scheduling — are now standard infrastructure [27]. None of them asks who is on the other end of the stream. One preprint measures what that costs when nobody is watching the stream, by comparing SLO-constrained serving against an unconstrained baseline with no latency targets at all.

The longer the context, the more the guarantee costs.

The measurements are on Qwen-2.5-32B in FP16 on an 8-GPU H20 node, against a closed-loop benchmark of 1000 sessions of 10 rounds derived from Mooncake ToolAgent traces [27]. What is being priced is the latency constraint, not the KV residency.

The same paper runs the human-interactive case as a control, at a different SLO setting. On Poisson arrivals with a 2-second mean inter-arrival time, at 64K context and production SLO settings, the sacrifice is 1.9%, at 6030 versus 5920 tokens per second [27]. The tax is not the cost of caring about latency. It is the cost of caring about latency for a caller who is a loop.

"Human-less serving scales with concurrency; SLO-constrained serving does not." [27]

— Human-less LLM Serving

Set that against the trace's 92.3% [1] and the shape of the error is the same one the hit rate has: an instrument that averages over the wrong population. The correction is not "stop caring about latency," and two results mark the boundaries.

PASTE speculatively executes predicted tool calls while the model is still generating, reducing average task completion time by 43.5% and observed tool latency by 1.8× [28]. It has to jointly schedule tool execution against returning LLM sessions, specifically to avoid shifting the bottleneck onto the GPU [28]. Accelerating one phase of a loop relocates the queue rather than removing it. From the queueing side, a class of work-conserving schedulers is provably throughput-optimal both for individual requests and for agent workloads with DAG and fork-join routing — with Orca and Sarathi-Serve confirmed throughput-optimal and FasterTransformer and vanilla vLLM shown not to be maximally stable — but the same analysis notes that cyclic routing topologies complicate the throughput picture [29]. The agent loop is cyclic. "Just keep the GPU busy" carries that asterisk.

So what kind of workload is this?

The literature's characterizations of agent traffic look contradictory. The resume step is what reconciles them.

The prefill-heavy reading has a specific source: CacheWise reports that coding-agent requests show roughly a 21× higher ratio of prefill to decode tokens than chatbot workloads [30]. Read the condition attached to it and the number changes meaning — it is the accounting for what each request must prefill without KV-cache sharing across turns [30]. That is a counterfactual, and an honest one. CacheWise's own contribution is prefix-aware scheduling with reuse-aware eviction, which cuts KV cache evictions by up to 2–2.6× [30]. The figure is real; the condition attached to it is what makes it usable.

Production sits nowhere near that end. In the trace, prefill amplification is 5.3× against an eviction-free ideal of 1× [3]. The distance between 21× and 5.3× is not a disagreement about what agents are; it is the difference between an accounting with cross-turn sharing switched off and a fleet running managed caches with it switched on.

A third measurement moves the axis instead of the value. An independent characterization running Claude Code against Gemma and Qwen models on vLLM across five benchmarks finds execution primarily LLM-dominated, with inference taking more time than tool execution across every benchmark and model tested [31]. The model's own behaviour moves the workload as much as the serving stack does. On one benchmark, Gemma in instant mode averages 108.8 ± 178.7 turns against 18.0 ± 15.3 for the same model in thinking mode, with a tail reaching 786 turns, driven by a pathological retry loop of 2,757 edit attempts at a 95.4% failure rate [31]. Context that long is not a property of agents. It is a property of an agent failing.

Three different units, then — a token ratio per request, an amplification factor, a share of execution time — and they cannot be chained arithmetically. They are bound by a shared cause rather than a common scale. The synthesis is ours, not any paper's: agents are not inherently a prefill-heavy workload. They are a workload whose phase is set by whether accumulated state survives the gap to the next step.

The bill follows the same logic and inverts the usual intuition about where money goes. Across the trace, prefix tokens account for 61.7% of cost, at an average of $9.36 per session [3]. That is the majority of the spend, even though those tokens are billed at roughly a tenth of the fresh-input rate. Generation, the part everyone watches, is the cheap part.

And then the layer above can void all of it. Compaction — the harness deciding the context has grown too long and rewriting it — affects 9.7% of sessions in the trace [3]. Of those compactions, 86.5% are tool-initiated rather than requested by a person [3]. NVIDIA describes the same operation from the serving side: context summarization compressing roughly 175K tokens down to about 40K [16]. Reasoning blocks that account for around 40% of generated tokens become ephemeral the moment the reasoning loop closes [16]. The server can hold the prefix as long as it likes; the harness decides when the prefix stops existing.

"The biggest optimization surface in agentic inference is the gap between what the harness knows and what the infrastructure can see." [16]

— NVIDIA, on Dynamo

One negative result is worth stating, because it is the assumption most people bring to this topic. Nothing in these traces branches. They describe linear, append-dominant context growth punctuated by compaction resets — no forked histories, no speculative exploration of alternate conversations. The only fork mechanism among the sources reviewed here is ForkKV's copy-on-write design, motivated by multi-LoRA serving rather than observed in a trace [21]. The only branch-shaped object anyone measured is subagent spawning, and it is a vendor measuring someone else's managed service. NVIDIA reports teammate agents averaging a 79.4% cache hit rate against 91.3% for a lead agent's explore subagents, at read-to-write ratios of 5.0× and 11.7× [16]. Branching may well be where agent serving goes. On this evidence, it is not where it currently is.

Which leaves a smaller and more actionable claim than "optimize the cache." The instruments the field inherited from chat — a cumulative hit rate, a time-to-first-token target, a prefill-to-decode ratio — each describe the fleet, and the fleet is dominated by calls that were never at risk. The call that is at risk is the one after the gap. Every mechanism above is a different way of deciding what happens to state during a pause: how long to keep it, where to keep it, which tier to keep it in, and who gets to say when it dies.