The Memory Bill Comes Due: HBM and the Economics of Inference

Why did memory, not compute, become the constraint?

Scope first, because this thesis is easy to overclaim. This analysis concerns large autoregressive models served on modern accelerators under decode-heavy loads; it does not claim that memory dominates every inference workload. Long-prompt, short-answer jobs lean on compute during prefill. Small models at low batch sizes may never saturate the memory bus. Mixture-of-experts systems can bottleneck on expert routing and communication. Multi-node serving can be governed by interconnect latency rather than HBM. Edge inference runs under a different cost model entirely. Within its stated scope, though, the evidence stacks up in one direction.

Start with the physics, because the economics are downstream of it. A widely cited hardware-scaling analysis by Gholami et al. (the AI and Memory Wall paper, published on arXiv) puts peak server FLOPS growth at 3.0x every two years over the past 20 years, against 1.6x for DRAM bandwidth and 1.4x for interconnect, while model parameter counts scaled 410x per two years over 2018-2022 [1]. Each side of that gap is an exponential. The gap between them is too.

Training mostly hid this. Inference exposes it, because inference has two phases with opposite personalities. Prefill, chewing through the prompt, is one big batched matrix multiply. An engineering write-up of a production deployment (measured on that one stack, not a general law) reports prefill achieving 200-400 arithmetic operations per byte of memory accessed and 90-95% tensor core utilization on an H100 running a 70B-class model [2]. Decode is the opposite: one token at a time, the working set streamed from HBM on every step, with compute utilization in the same deployment cratering to 28-30% while the memory bus saturates [2]. Below the roofline's ridge point, the GPU's arithmetic units are just expensive spectators.

Here's the tell, and this time with its conditions attached. NVIDIA's H200 and H100 share compute throughput — 1,979 TFLOPS of FP8 in one provider's side-by-side spec sheet [3] (NVIDIA's own page quotes 3,958 TFLOPS for the H200 [4]; the doubled figure counts structured sparsity, a qualifier we add, not the sources) — and differ almost entirely in memory: 141 GB of HBM3e at 4.8 TB/s versus 80 GB of HBM3 at 3.35 TB/s [3, 4]. NVIDIA's launch benchmark (vendor-reported, so read it as a best case) states that "an offline summarization scenario (ISL/OSL=2048/128) with Llama-70B on a single H200 is 1.9x more performant than H100" [5]. The conditions matter: TensorRT-LLM v0.5.0, max-throughput mode, a 2,048-token input and 128-token output, single GPU (FP8 at batch 64, per one provider's account of the same benchmark [3, 5]); the same document shows the same chip spanning 341 to 3,014 output tokens per second as batch size and sequence shape change [5]. Be careful what the 1.9x proves: a 2,048-in, 128-out summarization job is a long-prompt, short-output shape, the row this article's own decision table assigns to prefill compute. NVIDIA's own gloss names the actual mechanism: "Max throughput improves due to its dependence on memory capacity and bandwidth, benefitting from the new HBM3e. First token latency is compute bound for most ISLs, meaning H200 retains similar time to first token as H100" [5]. The prefill term didn't speed up; the larger, faster memory let the chip hold and stream a bigger batch. Same math engine, up to nearly double the tokens, delivered through memory capacity and bandwidth. If compute were the constraint, that number would be impossible.

An independent technical write-up runs the decode arithmetic to its ceiling: a 70B-parameter model in FP16 on an H100's 3.35 TB/s of bandwidth tops out around 24 tokens per second per stream, because the weights simply cannot be read from memory any faster [6]. (An idealization: 140 GB of FP16 weights will not fit one 80 GB card, so this is a per-GPU-bandwidth ceiling, not a single-card setup.)

And then there's the KV cache, the tax that grows with context. Every generated token requires re-reading the cached keys and values of everything before it, and the cache grows linearly with sequence length and batch size. A practitioner guide calculates that a single Llama 3.1 70B request at 128K context consumes roughly 42.9 GB of GPU memory for KV cache alone at BF16 [7]: more than half an 80 GB card gone before you serve a second user. Long context, the feature every product roadmap promises, is a memory-capacity feature wearing a software costume.

So the constraint isn't that GPUs got slow. It's that the thing tokens are made of — bytes per second out of HBM — is the one input that scaled worst. The next question is what that costs, in dollars.

What does a generated token actually cost?

The metaphor (the price of a token is the price of moving bytes) is only useful if you can turn it into arithmetic someone could check. Two evidenced relationships do most of the work.

First, the physical layer. During decode, throughput is approximately effective memory bandwidth divided by the bytes that must be read per generated token: the model weights (divided across the batch) plus each request's KV cache. This is the standard memory-bound roofline framing, and Epoch AI's inference-economics model — an arXiv preprint by Ege Erdil (2506.04645, June 2025), not peer-reviewed — formalizes token latency as the maximum of memory-read time and compute time [9]. The same preprint offers a concrete anchor: reading a full 80 GB of HBM at 3.3 TB/s takes about 24 ms per token, a theoretical limit of roughly 42 tokens per second on an H100 [9], the same mechanism behind the ~24 tokens-per-second ceiling for a 70B-parameter FP16 weight set cited above [6].

Second, the economic layer. SemiAnalysis's InferenceX glossary states the conversion exactly: $/M = TCO($/GPU-hour) × 1,000,000 / (3600 × tok/s/GPU) [10]. Cost per token is the GPU-hour price divided by the tokens you actually get out of the hour. Every operator lever in this piece attacks one of those two terms — more tokens per second out of the same silicon, or cheaper hours.

Both component types are observable. On the denominator side, GPU-hour prices are market data: H100 on-demand rates settled around $2.85-3.50 per hour by late 2025, down 64-75% from 2024 peaks according to one market tracker [11]; one-year H100 contracts then climbed to $2.35 per hour by March 2026, a nearly 40% rise from $1.70 in October 2025, per market-index tracking [12] — measured prices, though from trackers with an interest in the rental market looking dynamic. On the numerator side, tokens per second is measurable per workload, and it is nowhere near the spec sheet.

That gap deserves its own vocabulary, because vendors quote the top of a four-rung ladder and operators live at the bottom. Peak bandwidth is the interface spec. Attainable bandwidth is what a real controller sustains — one industry technical comparison lists HBM3E at 1.15 TB/s peak per stack against roughly 1,100 GB/s sustained once refresh overhead is paid [13]. Application throughput is tokens per second after batching, parallelism, scheduling, and KV layout take their cuts — NVIDIA's own launch tables run from 341 to 3,014 tokens per second on identical silicon depending on workload shape [5]. And economic throughput is tokens per dollar after utilization and idle time — Epoch's model adds that serial generation speed scales with only the cube root of memory bandwidth [9], which is the mathematical way of saying you cannot buy your way to fast tokens with hardware alone.

What the simple model deliberately omits: host and networking costs, scheduler efficiency, idle capacity, and the prefill/decode mix. SemiAnalysis's own glossary warns that workload shape, interactivity, and utilization determine whether two cost-per-token numbers are even comparable [10]. It's a framework for checking claims, not for settling them. But it's enough to see why the rest of this piece is about who sets the price of bytes.

Who captures the margin when memory is scarce?

Three companies make HBM, and next year's supply at each is described as spoken for. This time, the receipts: supplier by supplier, each a company statement made during an annual price-negotiation cycle, which is exactly when "sold out" is the most profitable thing to say.

Samsung, on its Q3 2025 earnings call: "We've significantly expanded our HBM production for next year compared with this year, yet customer demand has already outpaced supply," per Kim Jae-june, executive vice president of the memory business — with preorders for 2026 already exceeding planned output, in a quarter where memory revenue hit a record 26.7 trillion won [15]. Micron's entire 2026 HBM supply is reported sold out under fixed-price contracts [16]. SK hynix had finished its 2026 price negotiations by October 2025 [18], and allocation trackers list HBM3E as effectively sold out for 2026, with prices up double digits year-over-year [26]. Company-reported for two suppliers, tracker-estimated for the third — but no longer an unsourced everybody-is-sold-out.

Market share is softer than it sounds. Counterpoint Research put Q3 2025 HBM share at 53% for SK hynix, 35% for Samsung, and 11% for Micron [19], while a separate industry write-up citing Counterpoint for the same quarter reports 57% and 22% for the top two [20], and a Micron-focused analyst note credits Micron with around 21% by early 2026 [16]. Three numbers for the same market, at the time of publication of each. When a product is allocated rather than sold, "share" is an estimate of private contracts, and the spread between estimates is the honest error bar. Directionally, TrendForce's estimate has SK hynix climbing toward roughly 70% by Q1 2026 on HBM strength [20].

Where does the money land inside the chip? Analyst teardowns (Silicon Analysts, drawing on TrendForce, Raymond James, and SemiAnalysis data; estimates throughout, since none of these firms sees invoices) put HBM plus advanced packaging at 60-70% of the bill of materials for leading accelerators; an H100 at an estimated $3,320 to manufacture against a roughly $28,000 average selling price; a GB200 superchip at about $13,500 to build, of which $5,800 is HBM, $2,200 packaging, and $1,700 logic [21]. Per-stack pricing in the same tracker moved from roughly $200 for HBM3 to $300 for HBM3E, with HBM4 estimated near $500 per stack as of July 2026 and HBM commanding a 5-6x premium over equivalent DDR5 capacity [21, 22]. The memory in one NVIDIA superchip, on these estimates, costs more than the entire manufacturing of the previous flagship.

Who keeps what, layer by layer? The margins below mix company-reported financials with analyst estimates — badges in the last column.

Read the table top to bottom and a pattern falls out: margins are fattest where supply is most concentrated, and they thin as you approach the customer, where a hundred-plus clouds compete on price. Scarcity rent pools upstream.

The second choke point is packaging. HBM doesn't ship in a socket; it's bonded next to the GPU die on a silicon interposer, which for NVIDIA means TSMC's CoWoS lines. Analyst tracking puts 2026 CoWoS demand near 1.0 million wafers, up from about 370k in 2024, with NVIDIA alone holding an estimated 60%, roughly 595,000 wafers [26, 18]. TSMC's CEO C.C. Wei described CoWoS capacity as "sold out through 2025 and well into 2026" [18], and the company is scaling from about 35,000 wafers per month in late 2024 toward 130,000 by end of 2026 [18, 27]. One report — a single-source claim, so it suggests rather than establishes — has Google cutting its 2026 TPU production target from about 4 million to 3 million units for want of CoWoS access [27]. Reserving the substrate competitors also need looks less like a side effect of NVIDIA's strategy than a load-bearing part of it — an inference from the allocation pattern, not a documented intent.

One more mechanism ties the memory market together, in the words of Micron's business chief Sumit Sadana: "As we increase HBM supply, it leaves less memory left over for the non-HBM portion of the market, because of this three-to-one basis" [20]. Each HBM wafer displaces roughly three wafers of conventional DRAM. Scarcity in the premium product manufactures scarcity in the commodity one. Which is where the pricing story gets strange.

Does the margin really stack twice?

The prior version of this piece said the margin was captured twice: once by the memory oligopoly, once again by NVIDIA's markup on top of it. That's the right instinct and too simple an accounting, so here is the claim rebuilt with badges on.

The upstream leg is a sell-side forecast. Bernstein projects HBM average selling prices rising 2 to 2.5 times by 2027 [28]. Its argument runs through a genuine oddity: between Q3 2025 and Q2 2026, standard DRAM prices rose roughly 4.5 times while HBM contract prices barely moved, locked by annual agreements — to the point where, per the report, "To match the revenue per wafer of conventional DRAM, HBM prices would need to triple" [28]. The premium product currently earns less per wafer than the commodity. That's the memory makers' case for the coming hike, made by analysts whose price targets rise with it: the same report lifts 2027 EPS estimates 26% above consensus for Samsung, 32% for SK hynix, and 38% for Micron [28]. Interest flagged; the wafer-economics observation is still a real observation.

The downstream leg is a model, not a measurement. A companion analysis estimates that a 2-2.5x HBM increase, amplified through GPU-vendor markups, lifts an NVIDIA Vera Rubin rack's selling price by about 12% and — combined with DRAM and NAND increases — cloud providers' AI infrastructure costs by roughly 30% [29]. The mechanism is stated openly: HBM was only about 5% of that rack's price before the hike, and the 12% figure assumes NVIDIA marks the cost increase up roughly fourfold to preserve a 75% gross margin and passes it on fully [29]. Every word after assumes is doing work. Real pass-through is diluted by yields, R&D amortization, system components, warranty, channel, and demand elasticity — none of which appear in the multiplication. So: the margin stack is a real direction with an uncertain magnitude. Suppliers protect gross margin percentages, so input-cost increases tend to be amplified in list prices; how much of that survives negotiation with hyperscalers who are also building their own chips is exactly the number nobody outside those meetings has.

What the buyer at the end of the chain can verify is the table in the previous section: at the time of publication, every upstream layer reports margins that would be extraordinary in any other hardware market, while the serving layer fights over rental prices a few dimes apart. That asymmetry — not any particular forecast — is the evidence that pricing power currently lives upstream. Whether it stays there depends on the next generation of memory.

Will HBM4 buy the industry out of the wall?

The suppliers' answer is HBM4, and the honest reading is: it buys time, not escape — and the time is already pre-sold.

The saga so far, dates and all. SK hynix announced completed HBM4 development on September 12, 2025 — a vendor claim from its own newsroom — with 2,048 I/O terminals, double the previous generation's interface width, operating speed above 10 Gbps against a JEDEC standard of 8 Gbps, and a claimed power-efficiency improvement of more than 40% [30]. Coverage from CES 2026 has SK hynix targeting volume production in the third quarter of 2026, with per-device bandwidth above 2 TB/s [32]. Samsung, for its part, told investors on its Q4 2025 call that "The Memory Business is on track to begin delivering HBM4 products this quarter — including those with industry-leading 11.7 Gbps performance" [33], and one industry account has Samsung shipping its first commercial HBM4 on February 12, 2026, taking roughly one-third of NVIDIA's Vera Rubin allocation against SK hynix's two-thirds [20], consistent with a separate analyst estimate of a ~62% SK hynix HBM4 share [26].

But HBM4 doubles the bandwidth needs the four-rung ladder from the cost-model section, because each rung takes a cut. Peak: an industry comparison projects 2.0-2.5 TB/s per stack, a 75-120% increase over HBM3E's 1.15 TB/s [13] — projections, not shipped-part measurements (the announced 10-11.7 Gbps pin speeds [33] on a 2,048-bit interface would imply roughly 2.6-3.0 TB/s per stack; author inference from the reported figures). Attainable: the same source lists HBM3E sustaining roughly 1,100 GB/s of its 1.15 TB/s peak after refresh overhead [13], and there is no public equivalent measurement for HBM4 yet — a genuinely open number. Application: tokens per second depends on the memory controller, parallelism, scheduler, and KV layout; NVIDIA's own benchmark tables show an order-of-magnitude spread on a single chip across workload shapes [5]. Economic: SemiAnalysis benchmark data shows what capacity is worth when it binds — the GB300's 288 GB per GPU, against the GB200's 192, unlocks serving configurations that lift performance-per-dollar 2.31x despite a 20% higher per-GPU cost [10]. Vendors will quote rung one. Your bill is set on rung four.

Meanwhile the capacity arms race is naked on the spec sheets. NVIDIA's B200 carries 192 GB of HBM3e at up to 8 TB/s [34]. AMD's answer is blunter: the MI355X stacks 288 GB at 8 TB/s, generally available since October 2025, following the MI300X's 192 GB and the MI325X's 256 GB [35]. Amazon's Trainium3 ships with 144 GB of HBM3e at 4.9 TB/s (1.7x its predecessor's bandwidth) and scales to UltraServers carrying 20.7 TB of HBM [36]. A TechInsights projection has memory demand from custom ASICs growing 62.5% in 2026 versus 42.8% for GPUs [37]. The hyperscalers designed their own chips partly to escape NVIDIA's margin; they cannot design their way out of SK hynix's, because every credible inference chip converges on the same three memory vendors. The exceptions prove the rule: Cerebras-style wafer-scale SRAM offers enormous on-die bandwidth but only 44 GB of capacity (room for models up to about 22 billion parameters in FP16 [6]) — a niche, not a successor.

So HBM4 widens the pipe, on vendor and analyst numbers, by something like double at the interface. The wall moves. It does not fall — and every meter it moves is billed at an estimated $500 a stack [22] by an oligopoly that pre-sold the capacity before the parts existed. The operator's problem is living profitably against that bill.

Which lever should an operator pull — and for which workload?

The previous edition ranked optimizations by leverage. Buyers asked a better question: what does my workload need? Here is the same evidence, reorganized the way the decision actually presents itself.

Three notes that don't fit in a table.

Batch size moves you along the roofline. One provider's analysis puts batch sizes of 1-8 deeply in memory-bandwidth-bound territory, while 128+ shifts the workload toward compute-bound and narrows the H100/H200 gap [3]. The same workload can sit on either side of the ridge depending on how you schedule it, which is why is inference memory-bound? has no workload-free answer — and why this article's scope sentence exists.

Buy the ratio, not the biggest number. One provider's benchmark makes the spread vivid: a B200 running Llama 3.3 70B in FP4 at ~10,000 tokens per second against an H200's ~2,500 in FP8 works out to roughly $0.17 versus $0.50 per million tokens [40]. Conditions attached: one model, different quantization formats on each chip, one provider's rental prices — a shape, not a universal ratio. And spec-sheet advantages must survive contact with the rental market: per SemiAnalysis's analysis, for sub-6-month rentals NVIDIA wins performance-per-dollar essentially always, because 100+ competing neoclouds discount NVIDIA hardware while scarce AMD supply stays artificially priced [8]. AMD's memory-per-dollar is a spec-sheet fact; whether you can rent it at a price that monetizes the advantage is a market fact, and at the time of publication the market says mostly no.

The savings decay; the input costs don't. Every optimization above is available to your competitors, and the market reprices accordingly. Aggregated pricing trackers — estimates assembled from list prices, so directionally solid and precisely soft — put GPT-4-class capability at $30 per million input tokens and $60 per million output in March 2023, versus around $0.40-0.80 per million by March 2026 [14], with one index of frontier output prices down roughly 94.5% since March 2023 [17]. The same trackers expect the decline to slow from roughly 10x annually toward 3-5x annually through 2027 as the easy waste runs out [14]. Meanwhile inference has grown to an estimated 67% of all AI compute demand in 2026, from roughly 33% in 2023 [31], one industry guide puts lifetime inference spend at 5-10x training cost for high-usage models [25], and Epoch reports inference revenue at major labs growing 3x per year [9]. Software efficiency became the baseline; the memory bill became the floor. The floor is the part that's rising.

What would prove the memory-business thesis wrong?

A thesis you can't falsify is a marketing narrative, and this one has beneficiaries — memory vendors negotiating annual contracts, sell-side desks with raised price targets, and yes, publications that get to write the memory bill comes due. So, on the record, here is what would break it.

Supply lands. Samsung plans to expand HBM production capacity by around 50% in 2026 [19]. If that lands on time and on yield — the yield gap is the open question, with one industry account putting SK hynix's stack yields at 75-80% against Samsung's 60-65%, analyst estimates both [20] — the scarcity premium compresses, and the margin stack argument deflates with it. Watch shipped bits, not announced capex.

Demand was narrative. The OpenAI-Samsung-SK hynix letter of intent for 900,000 DRAM wafers a month for Stargate [19] is an LOI, not a purchase order. If it quietly shrinks, the demand-doubling scenario that anchors the bullish HBM math shrinks with it. Similarly, the "sold out" statements from the suppliers [15, 16] were made during price negotiations; sold-out-at-what-price is the unreported variable.

The wafer arithmetic inverts. Bernstein's own observation, that HBM would need to triple in price to match commodity DRAM revenue per wafer [28], cuts both ways. If commodity DRAM stays this profitable, the rational supplier shifts wafers away from HBM, per Sadana's three-to-one displacement [20]. SK Group's chairman says "The wafer shortage will likely persist until 2030" [20] — a chairman of a memory conglomerate, describing a shortage of the thing he sells. Trend, not law; the DRAM cycle has humbled every straight-line forecast drawn through it.

Software outruns the wall. Per-token prices fell roughly 94.5% at the frontier in three years [17] while HBM ASPs were flat-to-rising [28] — software efficiency has, so far, beaten the memory bill in per-token terms. If quantization, sparsity, and speculative execution keep compounding (NVFP4 already cuts KV bytes 75% on Blackwell [7]), memory stays the largest hardware line item while mattering less to application economics every quarter. The thesis would survive as a statement about supplier revenue and fail as a statement about what constrains AI products.

The workload mix shifts. Everything here is scoped to decode-heavy serving of large dense models. A market that tilts toward prefill-heavy retrieval, small distilled models, or MoE architectures bottlenecked on routing and interconnect is a market where this article's constraint is the wrong one. The scope sentence in section one isn't a disclaimer; it's the falsification condition.

None of these have happened at the time of publication. Two of them, Samsung's ramp and the wafer arithmetic, have dates attached in 2026 and 2027. That's what makes this a checkable thesis rather than a vibe: the memory bill has come due before, in every DRAM cycle since the term existed, and the industry has always eventually printed more receipts than the collectors expected.