Tag
#serving
Reports tagged serving.
Reports
4 reportsAug 2026
08-04 Deep Dive The Cache Miss After the Gap Coding agents serve 95.7% of their input tokens from cache. The steps that miss cluster in one place — the first call after an idle gap — and that is where the serving work went. Jul 2026
07-29 Deep Dive Two Workloads, One Wire: What Splitting Prefill and Decode Actually Buys Splitting prefill from decode does not remove the contention between them. It relocates that contention onto a wire, and every layer built since exists to pay that bill down. 07-29 Deep Dive Every Token Is a Read: How Memory Bandwidth Prices AI Inference, and Who Collects the Rent An H100 is sold on its arithmetic, but emitting a token is mostly a memory read — and the firms that sell those bytes now book a richer margin than the company whose name is on the accelerator. 07-29 Deep Dive The KV Cache Became Inventory — and Almost Nobody Charges Rent Yet Serving stacks now give the KV cache its own flash tier, its own router and its own rack. The bill a customer sees still discounts recomputation avoided, not memory held.