The KV-Cache Economy

Why does a million tokens of context cost what it does?

The price of long context is, to a first approximation, the price of holding state in fast memory. During generation a transformer keeps a key-value tensor for every token it has seen, and that cache grows linearly with sequence length, layer count, and head dimension. The magnitudes are what make this an economic story rather than an implementation detail. A worked example for a 7B-class model (32 layers, 32 query heads, head dimension 128, BF16) puts the full multi-head-attention cache at 64 GB for a 128K-token context [1]. For Llama-3.1-70B, one million tokens of context requires 305 GB of KV-cache [2]. A vendor-adjacent engineering guide suggests roughly 40 GB per user at 128K for the same 70B model on an H100, so eight concurrent users would need around 320 GB — described there as 4x the H100's high-bandwidth memory [3]. The state does not fit where the compute is.

The memory it must sit in is also the steepest part of the hardware cost curve, with a bandwidth ladder that the same guide sketches as roughly 3.35 TB/s for GPU HBM, 63 GB/s for CPU DRAM, and 7 GB/s for NVMe [3]. Where that HBM comes from, and why it is scarce, is a supply-chain story we have told separately; likewise the practice of splitting prefill and decode onto different fleets, which this piece touches only where it meets the cache. What matters here is the consequence: every long-context request reserves a slab of the most expensive memory in the rack for as long as it runs.

Vendor price lists reflect this directly, though not uniformly. Google publishes an explicit long-context step: Gemini 3.1 Pro moves from $2 per million input tokens and $12 per million output tokens to $4 and $18 above 200K tokens, Gemini 2.5 Pro from $1.25/$10 to $2.50/$15 past the same line, and the legacy 1.5 Pro doubled past 128K [4]. A model-comparison tracker reports that, as of June 2026, Google, MiniMax, and Alibaba still price long context in tiers, while Anthropic, OpenAI, and DeepSeek quote flat per-token rates across the window [5]. That is a C-tier aggregation of vendor pages, so treat the roster as indicative rather than settled. The flat rates are recent: Anthropic billed requests above 200K tokens at a premium tier of roughly $10 and $37.50 per million input and output tokens against standard rates of $5 and $25, and removed that surcharge in March 2026 [6]. Before removal, one practitioner analysis described the schedule bluntly as "a step function with a hard threshold at 200K input tokens" [7].

Removing a surcharge does not remove the physics. The same analysis reports time-to-first-token at 500K context of approximately 3.5 seconds when the prefix is cached against 35 seconds cold [7]; the cost moved from the invoice into latency and into the provider's own capacity planning. The mechanics compound during generation as well, since decoding reads back the accumulated cache as it proceeds, so a resident long context taxes memory bandwidth long after prefill has been paid for. That migration, from a visible price tier to an internal memory-management problem sold back to developers as caching, is the through-line of everything below.

What are vendors actually selling when they sell you a cache?

A prompt cache is a storage product with a rate card: a write premium, a read discount, a lifetime, and a minimum object size. Reading the four major rate cards side by side shows how differently the same physical object — a saved KV prefix — can be productized.

Anthropic sells the most explicit version. Cache writes cost 1.25x the normal input rate for the default tier and reads cost 10% of it, with a 5-minute default lifetime and a 1-hour option at a higher write rate [8]; the 1-hour write multiplier is 2x [10]. On Sonnet 4.6 that means $3 per million input tokens base, $3.75 to write the 5-minute cache, and $0.30 to read it [9]; on the current Fable 5 flagship, $10 base, $12.50 and $20 for the two write tiers, and $1 to read [10]. Developers place up to 4 explicit `cache_control` breakpoints [9], with a minimum cacheable prefix of 512 tokens on the Claude API and 1,024 on Amazon Bedrock [10]. Two details reward attention. First, cache reads do not count toward the input-tokens-per-minute rate limit on current models [9], so the cache is quietly also a rate-limit product. Second, the arithmetic swings hard with cadence: a worked example of a 50-iteration agent loop over a 200K-token prefix costs $12.30 with the 5-minute cache against $100.00 uncached, while a 2-hour human-in-the-loop session over a 100K prefix costs $3.40 on the 1-hour tier but $18.75 on the 5-minute tier because of repeated expiries [10]. These are one blog's model-calculations (B-tier), but they follow directly from the published multipliers.

OpenAI historically sold the opposite shape: automatic prefix detection, discounted reads, and no cache-write fee — still true for model families before GPT-5.6. That last part has since changed at the frontier. OpenAI's own prompt-caching documentation now draws the line by model family: "Cache writes have no additional fee on models before the GPT-5.6 family," but "For GPT-5.6 models and later model families, cache writes cost 1.25× the uncached input rate," with discounted reads retained throughout and a default cache-retention TTL of 30 minutes [42]. The read discount itself is model-dependent, placed by a router-vendor document between 0.1x and 0.5x of the input price [11], with the 50% figure attested for the GPT-4o and o-series generation and Azure [8]. Caching triggers at a 1,024-token minimum with hits extending in 128-token increments, and a single prefix begins to overflow above approximately 15 requests per minute [13]. The strategically interesting move came with GPT-5.1's developer launch on November 13, 2025: a `prompt_cache_retention` option of `24h`, which works, in OpenAI staffer Steven Heidel's words, because "with 24h prompt caching we move the caches from gpu memory to gpu-local storage. that storage is not free, but we made it free since it moves capacity from a limited resource (GPUs) to a more abundant resource (storage). then we can serve more traffic overall!" [12]. Azure's documentation now lists in-memory retention — cleared within 5 to 10 minutes of inactivity, always within one hour — as the legacy mode, with `24h` the default for models newer than gpt-5.4 and no opt-out from caching at all [13]. That is the clearest statement on record that the KV-cache has become a managed storage tier: the vendor is doing cache-tier engineering internally and selling the result as retention.

Google splits the product in two. Implicit caching, on by default since May 2025 [14], discounts hits — $1.50 per million input tokens dropping to $0.15 cached on Gemini 3.5 Flash [15] — with per-model minimum input sizes documented officially: Google's context-caching docs list a 4,096-token minimum for Gemini 3.5 Flash and 3.1 Pro and 2,048 for the Gemini 2.5 models [43], though lower figures still circulating on trackers reflect previous thresholds since raised. Explicit caching adds a line item absent from the other three rate cards examined here: storage at $1.00 per million tokens per hour for Flash-class models [15, 16]. Context is literally rented by the hour, which makes Google's pricing the most honest expression of what the product physically is, and also the easiest to leave running by mistake. The discounts do not stack with batch pricing — the cache discount takes precedence over the 50% batch discount [16] — and one C-tier report suggests an anomaly where implicit hit rates drop for prefixes between 9,000 and 17,000 tokens [15]; we could not verify it independently.

DeepSeek is the existence proof for how far the curve can drop. Its context cache has been disk-backed since mid-2024, when DeepSeek's own API documentation announced "Context Caching on Disk technology" and set the cache-hit rate at $0.014 per million tokens [17, 44]; current official pricing on deepseek-v4-flash is $0.0028 per million tokens for a cache hit against $0.14 for a miss, and the deepseek-v4-pro tier runs $0.003625 against $0.435 [45] — hit/miss spreads of roughly 50x and 120x by arithmetic on those rates. A single-user bill reproduced in that analysis — 88.9M input tokens at a 98.07% hit rate totaling $0.642 — is C-tier anecdote, but it illustrates the source's thesis, which we would print on the cover of this piece if we could: "The price card is the menu. The cache hit rate is the bill." [17]. Disk-resident caching at this price is made far cheaper by multi-head latent attention, whose roughly 8x smaller cache is cheaper to store and reload; MLA stores a compressed latent vector instead of full keys and values — in Sebastian Raschka's phrase, "MLA is not 'shared heads.' It is 'compressed cache state.'" — an approach now found in DeepSeek, Kimi K2, GLM-5, and Mistral 3 Large [18].

What hit rate can you actually expect — and what breaks it?

Every discount above is conditional, and the condition is fragile: the cached prefix must match exactly from the first token. Changes to the rendered prefix — including timestamps, tool definitions, message ordering, serialization, and sometimes whitespace — can invalidate the reusable suffix after the first mismatch [20]. This makes the hit rate an engineering artifact, not a property of the workload.

The strongest first-party articulation comes from the Manus team, whose July 2025 essay made "Design Around the KV-Cache" its first lesson and argued that "the KV-cache hit rate is the single most important metric for a production-stage AI agent" [19, 20]. The reason is the shape of agentic traffic: input-to-output token ratios around 100:1 [20], and in some agentic workflows exceeding 100:1 [23], mean nearly all spend is context re-reading. Manus's tactical corollaries — keep the prefix stable, make context append-only, and "Mask, Don't Remove" tools rather than editing the definitions mid-session [19] — are all cache-preservation rules wearing prompt-design clothes.

The best public before-and-after we found is ProjectDiscovery's. Their security agent runs workflows averaging 26 steps and 40 tool calls; dynamic content early in the prompt held the cache hit rate to 7%, and "moving dynamic content out of the prefix to the tail" took one deployment from 7% to a 74% hit rate, with further work reaching 84% and a 59% cut in model costs, on 9.8 billion tokens ultimately served from cache [21]. This is a first-party social-media statement — flagged as such, with no fuller primary write-up located — but an independent engineering guide repeats the 7%-to-84% arc and adds the breakeven arithmetic: under Anthropic's 1.25x write premium, caching pays only above roughly 1.4 reads per cached write [22]. A low hit rate on a write-premium vendor is worse than no caching.

Self-hosters face a second failure mode the API user never sees: the load balancer. In llm-d's benchmark (8 vLLM pods on 16 H100s serving 150 simulated enterprise customers with 6,000-token contexts, cache demand at 73% of cluster capacity), prefix-cache-aware scheduling delivered a P90 time-to-first-token of 0.542 seconds, against over 31 seconds for approximate routing, at 8,730 output tokens per second [23]. Alibaba Cloud's run of the same scenario put random scheduling above 92 seconds P90 and described the gain as a 170x improvement [24]. The design is a controlled same-hardware comparison, which is why we grade it A and let it carry causal weight: routing, not model or hardware, produced the difference. Both publishers ship competing cache-aware routers, so the framing is vendor-interested even where the measurement is clean.

The counterexamples matter as much. A May 11, 2026 vLLM benchmark on Qwen3.6-35B-A3B across 8 H800s recorded one prefix-caching mode at a 0% hit rate, and another achieving 54.4% hits yet lower throughput than caching disabled — 125.17 against 140.49 tokens per second [25]. And vLLM's own documentation is explicit that "APC only reduces the time of processing the queries (the prefilling phase) and does not reduce the time of generating new tokens (the decoding phase)" [26]. Caching buys back prefill, nothing else; decode-heavy workloads should expect little.

Can you tier your way out — HBM to DRAM to flash?

Below the API line, the operative question of the last nine months has been whether a KV prefix is worth keeping when it no longer fits in GPU memory — and the measured answer, for reuse-heavy workloads, is yes, even from disk. StorageReview's July 22, 2026 test is the cleanest public datapoint: a Dell PowerEdge XE7740 with four RTX PRO 6000 Blackwell GPUs (96 GB each), 1 TB of DDR5, and eight Solidigm PS1030 12.8 TB drives in RAID10, running vLLM 0.22.0 with LMCache 0.5.0 and MiniMax-M2.7. Against a VRAM-only baseline that saturated and fell into recompute, KV offload to DRAM delivered up to 2.9X total serving throughput (a 188% increase) and offload to flash 2.2X (a 122% increase) [27]. The authors' framing is the right one: "Past a certain context length, the decision is not fast DRAM versus flash; it's flash versus discarding the context, which is the eviction that costs throughput and latency in the first place." [27] Those multipliers should not be generalized to decode-heavy, low-reuse, or bandwidth-constrained workloads: the gains are a prefill-reuse story, and vLLM's own documentation is explicit that prefix caching saves prefill only [26].

The pattern predates the benchmark. Mooncake, the serving platform behind Moonshot AI's Kimi, built its architecture around a disaggregated KV-cache pooled across CPU, DRAM, and SSD resources (the same paper separates prefill from decoding clusters) and reports up to a 525% throughput increase in simulated scenarios and 75% more requests served under real workloads [28]; these are the operator's own numbers (vendor-authored, and the simulated figure is marketed rather than independently measured). NVIDIA's Dynamo formalized multi-tier offload for TensorRT-LLM and vLLM, with partner VAST reporting 35 GB/s of storage throughput into a single H100 [29] — again vendor-reported. IBM's Redbooks benchmark of a 120B model with cache persisted on Storage Scale claims a 56x time-to-first-token speedup at 130K-token inputs and a 22x throughput gain from 0.19 to 4.26 requests per second [30], vendor-measured on vendor hardware. The consistent direction across interested parties, together with the third-party StorageReview result (disclosed setup, vendor-supplied hardware), suggests the tiering effect is real even if individual multipliers deserve discounting. An llm-d filesystem backend adds a portability point: KV blocks stored as plain files on shared storage loaded up to 16.8X faster than re-prefill for long prompts [2].

The honest boundary conditions come from the tiering camp itself. LMCache's own AMD MI300X study of multi-turn agentic traces found its CPU/NVMe tier only wins once the sustained working set exceeds HBM capacity — a crossover around 250-300K tokens for that model-hardware pair, and that below it, plain HBM prefix caching was 10-17% faster; their summary: "There is no universal 'always enable LMCache' answer." [31] A simulation study across three production traces reaches the same edge from the cost side: "In contrary to DRAM, disk's limited bandwidth often makes KV reloading slower than recomputation." [32] Reload beats recompute only when reuse actually arrives before eviction and bandwidth holds.

The market has noticed regardless. LMCache's lead argues the object has become "a first-class data object with its own lifecycle, storage stack, and economic value", noting dedicated KV-cache storage offerings from WEKA, VAST, and DDN, NVIDIA's renaming of its inference-context storage push to CMX, and a first industry tutorial on KV-cache at GTC in March 2026 [33]. When storage vendors sell a product named after your intermediate activation state, the memory product framing has stopped being a metaphor.

How far can compression shrink the memory product?

FP8 KV has become a common production option on supported Hopper/Blackwell-class stacks, halving cache size vs FP16/BF16; NVFP4 pushes that to 4-bit, 75% smaller than BF16, though it requires Blackwell [3]. The research answer of the past year goes further: three to four bits per value, with real but narrow caveats — and, in the strongest post-hoc comparison reviewed here, quantization outperformed rank reduction at matched storage budgets, while trained-in latent compression such as MLA remains a distinct, competitive path. TurboQuant, from Google Research and NYU and published at ICLR 2026, compresses the cache at least 6x at 3-bit against an FP16 baseline with no training or calibration, reports up to 8x faster attention at 4-bit on H100s, and comes with an information-theoretic bound placing it within a constant factor of about 2.7 of the Shannon limit [34]. A 128K context that needed 4 GB of cache fits in under 700 MB at 3-bit [34]. Adoption has been fast for a one-year-old paper: a local-AI SDK shipped it in production on June 2, 2026, motivated by the observation that a 4B model at 262K tokens already carries an 8 GB FP16 cache, and reports a 96.0% RULER score at 4-bit against the 16-bit baseline [35]. A C-tier capacity model suggests the serving translation: a 70B model's 128K cache falls from 40 GB to 6.7 GB, and on a 2-GPU node the same model's concurrent capacity rises from about 2 to about 11 users at 32K context [36]. An independent open-source replication measured logit cosine similarities of 0.988 at 3-bit and 0.997 at 4-bit, while also finding decode running at half FP16 speed without a fused kernel [37]: compression ratios port easily, kernels do not.

The comparative claim is in the title of an April 2026 paper — "Quantization Dominates Rank Reduction for KV-Cache Compression" [38] — and the architectural counterpoint is that MLA, which is rank reduction built into training rather than bolted on after, achieves its 2-8x reduction while holding quality in the DeepSeek lineage's own ablations [1, 18]. The distinction that matters for a buyer: post-hoc rank reduction underperforms post-hoc quantization, while trained-in latent compression is what makes disk-priced context on MLA models so cheap.

Two caveats deserve more attention than they get. The first is safety: a systematic study across eleven instruction-tuned models found low-bit KV quantization can degrade refusal behavior while perplexity barely moves (Mistral-7B losing 15.2% of its refusals at only 1.03x perplexity), with no universal safe bit-width, and key-side quantization accounting for 76-102% of the damage in most models studied [39]. Cache compression is invisible in the metrics most teams monitor. The second is infrastructure: NVIDIA researchers note that eviction-style compression methods often depend on attention scores that FlashAttention does not expose, and that in paged-memory systems evicted tokens do not free memory until whole blocks empty [40], a reminder that a compression paper's memory savings and a serving system's are different quantities. Where the pieces do line up, the payoff compounds: an edge multi-agent system persisting 4-bit caches to disk reports reload at approximately 500ms against a 15.7-second re-prefill, and a cold-versus-warm TTFT of 172,096ms against 1,819ms for Gemma 3 12B at 32K [41].

What should a context-heavy application engineer for?

Treat the cache as the unit of design, because it is now the unit of pricing. Five commitments follow from the evidence above.

Meter the hit rate first. It is the variable everything else routes through, and it is routinely unknown. The DeepSeek analysis found real bills running far below list price purely on cache math [17]; one practitioner heuristic holds that "A hit rate below 50% on a long-context workload is a sign the breakpoint is in the wrong place." [8] Until the number is on a dashboard, no other optimization here can be evaluated.

Buy prefix discipline before capacity. Static content first, dynamic content at the tail, deterministic serialization, no timestamps in the system prompt [19, 20]. This is free, and it is the entire mechanism behind ProjectDiscovery's 7%-to-84% swing [21, 22]. Only after prefix discipline is exhausted does paying for a bigger window make sense — and note that window size and usable context differ: at 1M tokens, measured recall varies enormously between models, 76% against 18.5% on one long-range benchmark [7].

Match cache lifetime to request cadence. On write-premium vendors the tier choice is a bet on your own traffic: the 5-minute tier recoups in one read, the 1-hour tier in two [6, 10], and the worked examples above show a 5x-plus cost swing from picking the wrong lifetime for a human-paced session [10]. On OpenAI, 24-hour retention has made the bet mostly moot for newer models [13]; on Google, hourly storage fees mean an idle explicit cache is a meter left running [15, 16].

If you self-host, route before you tier. The largest measured wins in this survey came from cache-aware scheduling on unchanged hardware [23, 24]; tiered offload earns its complexity only past the working-set crossover, which for one measured configuration sat around 250-300K tokens of sustained context [31]. And verify the cache helps at all — measured configurations exist where it costs throughput [25].

Price the spread, not the rate. At the time of publication, filling a 1M-token window costs $0.14 on DeepSeek V4 Flash, $4.00 on Gemini 3.1 Pro, and $10.00 on Claude Fable 5 by one tracker's comparison, a 71x spread [5], and the cached-versus-fresh spread within a single vendor is of the same order as the spread between vendors. A context-heavy application's unit economics are a function of two numbers: the blended input price implied by its hit rate, and the input-to-output ratio of its loop. For agentic ratios near 100:1 [20, 23], the cache line on the invoice is the business model. That is the KV-cache economy: the context window stopped being a limit and became a product, and the teams that thrive under it are the ones that engineer for the bill, not the benchmark.