Column
Deep Dive
How does the machinery actually work? Reported investigations that follow one system from mechanism to money, with named actors and stakes.
Reports
9 reports · 2,721 sources read · 250 receiptsAug 2026
08-04 The Cache Miss After the Gap Coding agents serve 95.7% of their input tokens from cache. The steps that miss cluster in one place — the first call after an idle gap — and that is where the serving work went. 08-01 Distillation Mostly Reweights — Until the Teacher Adds Something New A weaker teacher can beat a stronger one when its token distribution overlaps the student's. In one 2026 study that overlap tracked successful distillation — but did not guarantee it. Jul 2026
07-31 Distillation Copies the Transcript, Not the Teacher A student model learns from a transcript, not from a mind — which is why capability transfers cheaply, why refusal quietly does not, and why the policy fight is aimed at the wrong part of the machine. 07-29 Containment Moved Downstream: How Deployed Agents Fight Prompt Injection by Giving Up Reach In-model defenses give much of their gain back once attackers study them. The cuts that hold sit on what the agent may do — they cost reach, and no test has yet matched the scale that broke the rest. 07-29 Two Workloads, One Wire: What Splitting Prefill and Decode Actually Buys Splitting prefill from decode does not remove the contention between them. It relocates that contention onto a wire, and every layer built since exists to pay that bill down. 07-29 Every Token Is a Read: How Memory Bandwidth Prices AI Inference, and Who Collects the Rent An H100 is sold on its arithmetic, but emitting a token is mostly a memory read — and the firms that sell those bytes now book a richer margin than the company whose name is on the accelerator. 07-29 The KV Cache Became Inventory — and Almost Nobody Charges Rent Yet Serving stacks now give the KV cache its own flash tier, its own router and its own rack. The bill a customer sees still discounts recomputation avoided, not memory held. 07-29 The Description Is Always Loaded: How Agent Skills Break Tasks Without Being Invoked Skill libraries lift an agent's average and quietly break tasks it already solved, and one channel of that damage runs through a line of metadata that sits in context whether or not the skill is opened. 07-27 Ternary LLMs Remove the Multiplier, Not the Cost Ternary weights really do remove the multiplier and shrink a model eightfold. What the compression ratio hides is where that cost comes back: in tokens, in kernels, in silicon.