The Great Unbundling of the GPU: How Prefill/Decode Disaggregation Became the Production Default

Why did one GPU stop being the right unit of serving?

Every LLM request has two lives. First, prefill: the engine ingests your whole prompt at once, a massively parallel matrix-multiply festival that saturates a GPU's compute units. Then decode: the model emits one token at a time, each step reading the entire accumulated key-value cache from memory while doing comparatively little math. One phase is compute-bound. The other is memory-bandwidth-bound. For years both ran interleaved on the same GPU, and each made the other worse.

The interference is not subtle. When a long prompt's prefill lands in a batch, decoding requests in that batch stall behind it — inter-token latency spikes exactly when a user is mid-answer. Reported utilization tells the story (tertiary survey, not a measurement of record): prefill runs GPUs above 90% while decode-heavy phases idle at 20–40% on Llama-70B-class deployments [1]. A large share of the fleet's silicon was being paid for and not used, phase by phase, millisecond by millisecond.

The fix sounds almost too literal: put the two phases on different GPUs and ship the KV cache between them. DistServe, published at OSDI 2024, formalized it and measured up to 7.4x more requests served (or 12.6x tighter latency SLOs) than colocated serving, on a 32×A100 testbed [2]. Microsoft's Splitwise, working the same idea concurrently, reported 2.35x throughput with the same resources [3]. The trick was reframing the objective from raw throughput to goodput: requests served within their latency targets. Once you optimize for goodput, each phase wants its own parallelism strategy, its own batch policy, its own hardware, and colocated serving stops making sense on paper.

That's the paper story. The production story ran ahead of it. Moonshot AI's Mooncake, the serving platform behind the Kimi chatbot, had been running a KVCache-centric disaggregated architecture at consumer scale and published it in June 2024: up to a 525% throughput increase in overloaded simulated scenarios, and 75% more requests handled under real workloads [4]. By the time it won Best Paper at FAST '25, Mooncake was processing over 100 billion tokens daily, carrying over 80% of Kimi's traffic, and handling 115% more requests on A800 clusters and 107% more on H800 clusters than its predecessor, at less than half the cost [5].

The unit of serving had stopped being a GPU. It had become a fleet with two job descriptions — and everything else in this story follows from the problems that creates.

Who shipped it — and when did it stop being research?

The disclosure that changed the conversation came from Hangzhou. In late February 2025, DeepSeek closed out its Open Source Week by publishing the internals of its V3/R1 inference system: prefill units of 4 nodes running expert parallelism at EP32, decode units of 18 nodes at EP144, an asymmetry only possible because the phases are separated [8]. Over one 24-hour window (February 27–28, 2025), the system processed 608 billion input tokens and 168 billion output tokens on H800 GPUs, averaging 73.7k tokens/s prefill and 14.8k tokens/s decode throughput, at a total operating cost of $87,072 for the day at $2 per GPU-hour (figures archived from a secondary summary [6] and press coverage [7], not from DeepSeek's primary post). Priced at R1 API rates, that traffic was worth a theoretical $562,027, the 545% cost-profit margin headline that made mainstream press [7]. Whatever you think of theoretical margins (nights, discounts, and free tiers all cut against it, as DeepSeek itself noted), the disclosure was a live production blueprint, and serving teams everywhere read it.

The frameworks arrived within weeks — already in flight, but landing in a conversation the disclosure had reframed. NVIDIA announced Dynamo at GTC in March 2025 [1], an orchestration layer above vLLM, SGLang, and TensorRT-LLM that routes prefill and decode to dedicated worker pools, with launch benchmarks claiming up to 30x per-GPU throughput for DeepSeek-R1 671B on a GB200 NVL72 rack [11]. Those are marketed numbers on vendor hardware in best-case configurations, not independent measurements, but the direction matched what neutral parties were finding. May 2025 made disaggregation open-source-reproducible: the SGLang team deployed DeepSeek with prefill/decode separation and large-scale expert parallelism across 96 H100s on 12 nodes, reporting 57,674 input tokens/s per node during prefill [9]. The same month, Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA launched llm-d, a Kubernetes-native framework with disaggregation as a founding feature [14].

Then adoption compounded. The receipts, in order:

• September 25, 2025: SGLang published GB200 NVL72 results — 3.8x prefill and 4.8x decode throughput gains over its prior baseline [17].
• October 2025: SemiAnalysis's InferenceMAX benchmark adopted SGLang as its default engine for DeepSeek models; the measured figures on GB200 NVL72 were 26k input tokens/s per GPU for prefill and 13k output tokens/s per GPU for decode [16]. Independent benchmark harnesses for disaggregated configs are themselves a signal: you don't build test infrastructure for exotica.
• December 17, 2025: vLLM's large-scale serving push reported a sustained 2.2k tokens/s per H200 on a CoreWeave InfiniBand cluster with wide expert parallelism and disaggregated serving [15].
• February 12, 2026: Mooncake joined the PyTorch Ecosystem, positioning its KV-cache transfer and storage engine as shared infrastructure beneath both SGLang and vLLM [46, 47].
• March 24, 2026: llm-d entered the CNCF Sandbox, with AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI signed on as supporters; its v0.5 benchmarks showed Qwen3-32B scaling to ~120k tokens/s across 16 H100s while holding near-zero TTFT where a baseline Kubernetes service degraded [14].
• GTC, March 2026: Dynamo 1.0 went GA, with CoreWeave, Nebius, Pinterest, and Together AI named as adopters and a claimed 30–50% cluster-utilization improvement in production workloads [12, 13]. Vendor claim, keynote-sourced, though Pinterest is not a company that deploys keynote demos.
• April 7, 2026: AMD's ecosystem shipped the MORI-IO KV connector for vLLM on MI300X. In its benchmark, run within a single 8-GPU MI300X node split 1P+1D, 73/100 requests met both latency SLOs versus 26/100 for a standard colocated deployment [19]. AMD's ROCm team had already measured 1P1D disaggregation on MI300X serving Llama 3.3 70B under 7.1X tighter SLOs at P95 and 13.2X at P99 in chatbot scenarios [18].
• July 6, 2026: AWS made Disaggregated Prefill and Decode generally available on SageMaker HyperPod, moving KV cache between GPU pools over EFA with GPU-Direct RDMA [23, 24], with a parallel llm-d-powered path benchmarked on GPT-OSS using four prefill pods against one decode pod [25].

Meta, LinkedIn, Mistral, and Hugging Face are running or building on vLLM's disaggregated architecture, with Meta's team contributing the production router [3]. When a technique is simultaneously the default in the two dominant open-source engines, NVIDIA's flagship inference product, a CNCF project, and a managed AWS service, "production default" is description, not prediction. What that consensus hides is everything the split made hard — starting with the wire.

What does it actually cost to ship a KV cache between GPU pools?

Here's the bill nobody puts on a slide. For a Llama-3.1-70B-class model, the KV cache runs 327,680 bytes per token, about 1.34 GB for a single 4,096-token prompt [3]. Keep TTFT under 500ms with a 200ms prefill, and that cache has to cross the wire in roughly 300ms, which works out to about 4.5 GB/s of sustained bandwidth for one request [3]. Now batch it: a batch of 8 such prompts on a 70B model is around 4.3 GB in flight, which takes approximately 344ms over 100Gbps Ethernet (longer than many TTFT budgets on its own), versus approximately 172ms over InfiniBand HDR at 200Gbps, versus near-instantaneous inside an NVLink domain at 600+ GB/s [32]. One production-engineering guide puts the failure plainly, and it's worth quoting because operators keep rediscovering it: "The most common failure mode in disaggregated deployments is underestimating KV transfer cost" [32].

This is why the transfer layer grew its own software category. NVIDIA open-sourced NIXL at GTC 2025 [31]: a vendor-agnostic point-to-point transfer library that abstracts the transport under one API (RDMA over InfiniBand or RoCE, TCP fallback, NVMe-oF, even S3-compatible object storage), and it now sits inside Dynamo, TensorRT-LLM, vLLM, SGLang, Anyscale Ray, and LMCache, with support for AWS EFA networking and non-NVIDIA accelerators [30]. Mooncake's Transfer Engine plays the same role in the SGLang world, pushing prefilled KV cache over RDMA and NVLink [46], and SGLang's stack accepts either backend [10]. The interconnect tax didn't disappear; it got a standard collection agency.

The more interesting engineering is in hiding the tax rather than paying it up front. Naive disaggregation transfers the whole cache after prefill finishes, so first-token latency is prefill time plus full transfer time; layer-wise asynchronous transfer instead ships each layer's KV as soon as it's computed, hiding early layers' transmission behind later layers' compute, which reduces the effective penalty to roughly the final layer's transfer [39]. Research systems go further: FlowKV restructures the KV cache's memory layout to collapse the number of transfer operations, cutting average transmission latency from 0.944s to 0.053s, a 96% reduction [40]. And the Beyond the Buzz study (an NVIDIA simulation analysis — flag the provenance) argues that existing datacenter bandwidth is generally sufficient for the transfer, because bandwidth requirements fall relative to compute as input length grows: prefill cost is quadratic in sequence length while cache size is linear [41].

So the wire is an engineering problem with known solutions — on the right hardware. That qualifier is doing enormous work. Every solution above assumes RDMA-class fabric or an NVLink domain, and it's worth saying who benefits: disaggregation turns "buy fewer, better GPUs" into "buy two differentiated fleets plus the interconnect between them." The architecture that saves you money per token also deepens your commitment to high-bandwidth networking, which is NVIDIA's moat as much as CUDA is.

Who decides where a request runs?

Split the fleet and you've created a decision problem that didn't exist before: for every incoming request, something has to choose a prefill worker, choose a decode worker, decide whether to split at all, and keep the two pools sized to each other. That something is now a product category.

llm-d's answer, built on vLLM's pluggable KV Connector API with NIXL underneath for transport [34], is an inference scheduler wired into the Kubernetes gateway: an Envoy proxy hands each request to an Endpoint Picker that runs a Filter → Score → Pick pipeline of pluggable policies — prefix-cache affinity to route requests toward workers that already hold their KV blocks, load-aware scoring that prefers endpoints with lower KV-cache utilization, and a profile handler that steers prefill and decode phases to their respective pools [35] (detail that traces to AI-generated repository documentation, so treat it as unofficial). The most consequential plugin is the one that decides whether to disaggregate: llm-d's prefix-based decider triggers remote prefill only when the non-cached suffix of a prompt exceeds a configured token threshold — short prompts and cache-warm requests run end-to-end on a decode worker [36]. AWS ships the same policy as a managed default, with HyperPod routing requests below a 4,096-token threshold straight past the split [24]. Disaggregation-per-request, not per-fleet.

SGLang's router does the equivalent job with a proxy that pairs prefill and decode servers under a workload-balancing scheme, plus the unglamorous machinery production demands: heartbeat checks between pools every 5.0 seconds by default, with workers marked failed after 2 missed beats [10]. And above routing sits the harder problem the DistServe authors only had to solve once, statically: rate matching — keeping prefill capacity and decode capacity in proportion as traffic shifts. NVIDIA's Dynamo Planner attacks it with an SLO-driven control loop that watches prefill queue depth and decode KV-cache load and rescales the pools; in a Microsoft-NVIDIA demonstration on Azure Kubernetes Service, coverage suggests the planner scaled a Qwen3-32B-FP8 deployment from one to two prefill workers under a 200-user traffic surge while holding a 500-millisecond TTFT and 30-millisecond inter-token latency target [38], though that account traces to vendor-adjacent republications, so treat the specifics as illustrative rather than audited. KServe's v0.17 release (March 13, 2026) folded the same ideas into its LLMInferenceService, autoscaling on KV-cache utilization and queue depth instead of CPU [37].

The pool ratios themselves have become a published design space: DeepSeek runs 4-node prefill units against 18-node decode units [8], AWS benchmarked four prefill pods to one decode pod on GPT-OSS [25], and Beyond the Buzz's simulations conclude that fixed ratios degrade substantially without dynamic rate matching [41]. The scheduler, in other words, is where disaggregation stopped being a topology and became a policy — and policies, unlike topologies, can be wrong in new ways every hour.

What do the measured numbers say, once you strip the marketing — and below what scale does colocation still win?

Tier the evidence and the picture gets more honest.

Tier A — peer-reviewed or production-measured. DistServe's 7.4x goodput on 32 A100s [2] and Mooncake's 115%/107% production gains at 100-billion-token daily scale [5] remain the strongest results in this story: one refereed, one operating a consumer product.

Tier B — self-reported but reproducible. SGLang's 96-H100 deployment [9], vLLM's 2.2k tokens/s per H200 on CoreWeave [15], and AMD's MORI-IO run, a single 8-GPU MI300X node split 1P+1D, where 73/100 requests met both SLOs versus 26/100 colocated [19], are engineering-blog numbers with open-source configurations; the InferenceMAX figures [16] add a semi-independent check. TaiChi's controlled comparisons (more on them below) sit here too, an unrefereed arXiv preprint whose regime measurements ran on a 4-node A100 cluster but whose broader sweep leans on a simulator, and they cut both ways [26]. This tier suggests the split earns its keep on fast fabric at multi-node scale, and, when the objective is strict latency SLOs rather than raw throughput, even within a single node.

Tier C — marketed. Dynamo's 30x on GB200 NVL72 [11] bundles disaggregation with a hardware-generation leap and expert-parallel routing; the disaggregation-attributable share is necessarily much smaller. NVIDIA's 30–50% utilization claim [12, 13] is more plausible precisely because it's less flattering. One tertiary survey estimates cost-efficiency gains of 15–40%, workload- and network-dependent [1] — a plausible bracket, but from an aggregation page, not a study, and it belongs in this tier. The measured sources bound the same territory from both ends: a near-tripling of SLO-compliant requests when the setup fits [19], a 20–30% degradation when it doesn't [42]. That's a lot of money at fleet scale. It is not 30x.

Now the threshold question, because the counter-evidence is specific. BentoML's inference handbook, from a team that deploys this for customers, reports that when the workload is too small or the setup untuned, disaggregation degraded performance by 20-30% in their tests [42]. A four-node experiment on DeepSeek-V2-Lite found the sharpest version of the failure: a 1-prefill/3-decode configuration delivered beautiful inter-token latency (10 ms at 4096-token inputs) while TTFT blew past 37 seconds because the single prefill node saturated, and a plain routing setup with no disaggregation at all beat every disaggregated configuration on raw throughput on that fixed cluster [43]. The authors' verdict, verbatim: "So, is disaggregated prefill/decode a silver bullet? The answer is clearly no — at least not under the conditions tested here" [43]. Beyond the Buzz, simulating hundreds of thousands of design points, draws the boundary as: disaggregation is most effective for prefill-heavy traffic and models above roughly 10B parameters [41]. One operator-facing guide suggests an even blunter heuristic: below about $50k a month in GPU spend, the complexity likely isn't worth it [33]; that's a C-tier rule of thumb, but it points the same direction as the measured failures.

Read together: the efficiency claim survives scrutiny at multi-node scale with fast interconnect and mixed-length traffic. Shrink any of those three conditions and the baseline you were trying to beat starts winning again.

What breaks when you run it — and how is research un-splitting the split?

Operational complexity is the cost that never makes the benchmark table. A disaggregated deployment runs two engine fleets with separate configurations, a router with its own failure semantics, heartbeat and fault-tolerance machinery between pools [10], and a scheduler that is itself a resource consumer — llm-d's endpoint picker needs roughly 35 CPU cores at 100 requests/s of routing load [35]. Debugging splits too: SGLang's docs note that prefill and decode workers must be profiled separately [10]. None of this is exotic to anyone who has run microservices; all of it is new to teams whose serving stack was, eighteen months ago, one process.

The failure modes are correspondingly new. Rate mismatch, pools sized wrong for the traffic, produces starvation: one phase saturates while the other idles, and TTFT or ITL collapses depending on which side starved [43]. Cache-miss storms hit disaggregated systems harder because a zero-hit workload maximizes transfer volume exactly when prefill is busiest [43]. And under balanced latency targets, pure disaggregation has a measured ceiling: TaiChi's controlled study on Llama-2-70B across a 4-node A100 cluster found that with a 6s TTFT and 100ms TPOT objective, pure disaggregation achieved 50% goodput and pure aggregation just 16% — both bad, for opposite reasons [26]. The same study measured the regimes cleanly: aggregation wins under tight TTFT with relaxed TPOT (97% versus 42% goodput), disaggregation wins under the reverse (98% versus 7%) [26]. Neither architecture dominates; the SLO decides.

A word on the baseline. Colocated serving usually means chunked prefill: slice a long prompt into fixed token budgets and interleave the chunks with decode steps. The hybrid literature treats it as the incumbent and attacks its core assumption: DuetServe notes that Sarathi-Serve-style schedulers assume linear layers dominate latency and rely on a fixed token budget, ignoring the compute-bound/memory-bound split between the phases [27] — and the linear-layer saturation point shifts from roughly 2,048 tokens on an A100 to 8,192 on an H100, so a budget tuned for one hardware generation is miscalibrated on the next [27]. MuxWise frames the dilemma directly: capping the token budget to guarantee decode SLOs is precisely what limits throughput [44]. Chunked prefill blunts interference rather than removing it; it anchors the small-fleet end of the argument without ending it.

Which is why the research frontier is busy hybridizing. TaiChi itself unifies both modes and improves goodput by up to 77% over the state of the art, cutting TTFT by up to 13.2x versus pure disaggregation, by shifting instances along a spectrum between the two [26]. DuetServe multiplexes both phases on one GPU by partitioning streaming multiprocessors, aimed squarely at deployments too small to split [27]; MuxWise does intra-GPU multiplexing with layer-wise prefill scheduling to fill the utilization gaps disaggregation leaves [44]. semi-PD splits the computation but unifies the storage — prefill and decode run as separate compute streams over one memory pool, eliminating the KV transfer entirely, and reduces average end-to-end latency per request by 1.27-2.58x on DeepSeek-series models [45]. And for MoE models, ExpertPlex observes that instance-level disaggregation duplicates over 95% of weights across pools (those are the expert parameters) and that sharing experts while disaggregating only attention beats standard disaggregation by up to 2.01x on goodput [28].

Note what these systems are not doing: none of them argues the phases are the same workload. They accept the diagnosis and reject the surgery — separate the scheduling of prefill and decode, share whatever hardware and memory you can get away with. That's a trend in the literature, not a law of the field; but it suggests the "default" the industry just standardized is one point on a spectrum the research community is actively re-exploring from both ends.

Does prefill deserve its own silicon — or will the cache hierarchy shrink it?

Once the phases are separate services, a heretical question follows: why do both run on the same chip? Prefill amortizes each weight read across thousands of prompt tokens, so raw FLOPS, not memory bandwidth, sets its ceiling. Decode is the opposite. Yet both were buying HBM, the most supply-constrained component in an AI accelerator.

On September 9, 2025, at the AI Infra Summit, NVIDIA answered with the Rubin CPX: a GPU designed for the context phase, prefill, delivering 30 petaflops of NVFP4 compute with 128 GB of GDDR7 instead of HBM at roughly 1.8 TB/s of bandwidth [20], plus a marketed, unmeasured 3x attention acceleration versus the GB300 NVL72 generation [21]. Compare the standard Rubin: 50 petaFLOPS of NVFP4 but 288 GB of HBM4 at 13 TB/s [22]. The CPX trades away most of the bandwidth to make compute cheap. The Vera Rubin NVL144 CPX rack pairs 144 CPX GPUs with 144 standard Rubins and 36 Vera CPUs for 8 exaFLOPs of NVFP4 compute [20, 21], targeted at late 2026 [20]. It's a chip whose entire thesis is that disaggregation is permanent — and NVIDIA attached a marketed projection that deserves your skepticism: 30x–50x ROI, "$5B in revenue from a $100M CAPEX investment" [21]. No independent party can verify that before the hardware ships, and it assumes monetizable demand for million-token-context inference at current prices, indefinitely.

Heterogeneity from below is the quieter version of the same bet. Once prefill and decode are separate services connected by a KV-cache wire, nothing says they must share a vendor: Wallaroo has proposed AMD MI325X for prefill and IBM's Spyre ASIC for decode in on-prem configurations [29], and one engineering guide notes the bandwidth logic — MI300X offers 5.3 TB/s of memory bandwidth against the H100's 3.35 TB/s, suggesting decode-side placement for AMD parts [33]. Every accelerator startup that could never win a whole-model benchmark now only has to win one phase.

But the sharpest challenge to dedicated prefill silicon isn't a rival chip. It's the memory hierarchy growing under the KV cache. The cache has become a first-class object with its own storage tiers — GPU HBM at ~3.35 TB/s, CPU DRAM at ~63 GB/s, NVMe at ~7 GB/s [48] — and a measured pecking order: for a 129K-token context, serving from GPU cache yields a 1.09-second TTFT, CPU memory 1.57 seconds, disk 7.20 seconds, and full recomputation 54.38 seconds [49]. From 8K tokens onward, even disk-loading beats recomputing the prefix by 5–7x [49]; recomputation only wins below roughly 1K tokens [49]. Production systems have industrialized this: Mooncake Store pools KV cache across the cluster for global reuse, with adopters including Alibaba Cloud, Ant Group, JD.com, Tencent, and Meituan [46], vLLM featured Mooncake Store for cross-instance sharing on May 7, 2026 [47], and llm-d's filesystem backend measured KV-loading speedups of up to 16.8X over cache-miss prefill on long prompts [50].

Follow the logic to its uncomfortable end. Every cache hit is prefill work that never happens — DeepSeek already reports a 56.3% on-disk KV-cache hit rate in production [8], and llm-d's router only invokes remote prefill for the non-cached suffix of a prompt [36]. The better the cache hierarchy gets, the less prefill exists to deserve its own silicon. NVIDIA is effectively betting that context growth outruns cache reuse; the caching layer is betting the reverse. Both are shipping.

What should an operator running N GPUs actually do?

The evidence above compresses into a decision framework, ordered by fleet size.

One node: let the SLO decide. If you're optimizing throughput or cost per token, don't split — the transfer tax and orchestration overhead eat the win at this scale, and the measured downside is real: 20-30% degradation in untuned small deployments [42]. This is the segment DuetServe-style SM multiplexing [27] and chunked scheduling exist for. But if the binding constraint is strict inter-token latency, intra-node disaggregation is measured to win: AMD's 1P+1D split of a single 8-GPU MI300X node nearly tripled requests meeting both latency SLOs versus colocated serving [19].

A few nodes, mixed traffic: split per-request, not per-fleet. Steal the policy AWS ships as a default: requests below a token threshold (HyperPod uses 4,096) skip disaggregation and run end-to-end on a decode instance; only long, cache-cold prompts take the split path [24]. llm-d's prefix-based decider implements the same idea with the threshold applied to the non-cached suffix [36]. This captures most of the win while capping the interconnect tax.

Tens of nodes with RDMA-class fabric: disaggregate with asymmetric pools, and automate the ratio. DeepSeek's 4-prefill-to-18-decode node structure [8] and AWS's 4-prefill-pods-to-1-decode-pod benchmark [25] bracket the design space, but the simulation literature is blunt that static ratios degrade as traffic shifts [41]; budget for SLO-driven rescaling (Dynamo's planner [38], KServe's variant autoscaler [37]) or accept rate-mismatch starvation as an operational risk [43].

Serving big MoE models: watch the hybrids before committing capex. Paying for over 95% duplicated expert weights across two pools [28] is the line item that looks obvious only in retrospect, and TaiChi's numbers say pure disaggregation is already suboptimal under balanced SLOs [26].

Long-context or agentic workloads: build the cache hierarchy first. A KV-offload tier is cheaper than a prefill pool, and from 8K tokens up, even disk beats recomputation [49]. Disaggregation composes with caching — but the ordering of investment matters, because every point of cache-hit rate shrinks the prefill fleet you'll need [6].

The one-GPU-does-everything era ended the way most architectural eras end: not because the old way stopped working, but because someone published the margin they were making by abandoning it. The next fight is already scheduled — whether prefill keeps its own silicon and its own fleet, or whether caches, hybrids, and un-split schedulers shrink the great unbundling back into software.