Distillation Mostly Reweights — Until the Teacher Adds Something New

Where the gradient actually lands

A 3B student learns more from LLaMA2-7B than from LLaMA2-13B or LLaMA2-70B. Practitioners have met this shape before and filed it under the capacity gap: past some point a teacher is too far ahead to be useful, and the optimal teacher is not the largest one [1]. It is a rule of thumb with no mechanism attached, which is why it has stayed a rule of thumb.

A 2026 dynamics study of on-policy distillation puts an instrument on it. Two teachers, one student — Qwen3-1.7B-Base — and the teacher that scores slightly lower on AIME 2024, AIME 2025 and AMC 2023 produces the consistently better student [2]. The paper's account of why is not about strength at all. At every generated token, the student has a top-k set of candidate tokens and so does the teacher; the fraction of that set the two models share is the overlap ratio. The lower-scoring teacher starts with a higher one.

That single quantity turns out to be where distillation does its work. In the paper's studied math-domain configurations, across successful runs, the fraction of each model's probability mass assigned to the shared top-k intersection runs 97%–99% throughout training [2]. Not most of it. Nearly all of it. The tokens where the two models disagree about what comes next are, in probability terms, almost empty.

The paper then runs the ablation that converts a correlation into a locus. It decomposes the student's top-k support into the overlap region and its complement and trains on each in isolation. Optimizing only the overlap recovers nearly the full benefit of that study's own student-top-k OPD baseline; optimizing only the non-overlap tokens is consistently weaker [2].

"OPD's primary optimization signal lies in reweighting probability within the overlap region rather than in tokens outside it." [2]

— Yaxuan Li, Yuxin Zuo, Bingxiang He and Jinqian Zhang, Rethinking On-Policy Distillation of Large Language Models

The dynamic is also self-sustaining. Both the full-support and overlap-only runs raise the overlap ratio steadily from about 72% to above 91%, while the non-overlap run first loses ground and only partly recovers [2]. Once a token enters the shared high-probability region and the teacher favors it, reverse-KL updates concentrate more mass on it, which pushes competing non-overlap tokens out of the student's top-k altogether. The region the optimizer works in grows because it is being worked in.

This is an unreviewed 2026 preprint, and among the sources reviewed here it is the only one that measures the mechanism at token level. It should be read as a measurement, not as a settled fact. But the measurement is specific enough to carry a model.

The model is this. Distillation is a redistribution of probability mass the student already carries. Teacher and student each occupy a region of the same space, and almost all of the gradient's purchase lies in the part they share. Every decision a practitioner makes is then a decision about distance: who you stand near, how you get there, how hard you compress once you have arrived, and which parts of the region you visit.

The teacher is a distance, not a score

If the gradient lives in the overlap, teacher selection stops being a ranking problem and becomes a matching problem. The teacher must be close enough for the gradient to reach — and different enough to offer something worth transferring. Three lines of work that share no apparatus land on the first half of that from three directions.

The oldest is a fitted law. Distilling a broad range of sub-3B language models, MiniMA finds that the optimal teacher scale grows linearly with the student scale, consistently across model architectures and data scales [1]. Applied at 7B, it produced the concrete result at the top of this piece: distilling the same student from LLaMA2-13B and from LLaMA2-70B yields worse students than distilling from LLaMA2-7B [1]. The larger teacher is worse, not just wasteful.

The second is a controlled sweep. Apple's distillation scaling laws vary student and teacher size and training tokens under IsoFLOP budgets and reproduce the U-shape directly: student cross-entropy falls as the teacher improves, reaches an optimum, and then rises again [3]. The reframing is what matters. The paper's parameterization locates the gap in learning capacity — the teacher's own cross-entropy and the tokens it was trained on — rather than in parameter count, of which the size ratio is a special case [3]. Two teachers of identical size trained on different data are not interchangeable, which is exactly what a size-ratio rule cannot express.

The third measures the same thing during training rather than before it. The overlap ratio is computable on your own run in the first hundred steps, and the dynamics paper's finding is that it dominates: even when the teacher achieves higher benchmark scores, "mismatched thinking patterns produce low initial overlap that training cannot fully recover" [2]. Successful runs show overlap climbing; failing runs show it stagnant from the outset. The paper keeps this a necessary condition rather than a sufficient one, and offers no threshold, no predictive accuracy and no cross-setting classifier: its second condition is that the teacher must also carry knowledge the student has not already acquired [2].

The three escalate in usable order: a ratio you can apply before you start, a law that tells you what the ratio is a proxy for, and a diagnostic you can read while the run is going.

All of this is logit-level distillation, where the student sees the teacher's output distribution. Nothing in this evidence establishes an optimal-teacher law for reasoning-trace distillation, the R1-style recipe where you fine-tune a student on a large model's generated solutions and never touch its logits. The linear scale rule is not transferable to that setting, and it should not be carried there. What does cross the boundary is the frame — both regimes are reweighting stories — never the constants.

Early mismatch, the dynamics paper reports, causes a loss of benefit that cannot be recovered later in training [2]. So what do you do when your teacher and your student start far apart?

On-policy is not the other fork; it is the second leg

The standard framing of the sampling decision is a fork. Off-policy distillation trains the student on sequences the teacher generated; on-policy distillation samples sequences from the student and has the teacher score them, which closes the gap between what the student trains on and what it will actually produce at inference.

Framed as a fork, it invites a cost comparison, and one comparison travels further than the rest: on-policy distillation reaches a target for an order of magnitude less compute than off-policy. That claim is worth examining, because the three on-policy results usually cited for it all quietly begin somewhere the fork framing does not admit.

GKD, the canonical academic method, states it as a baseline convention: "All methods including baselines start from the supervised fine-tuned student checkpoint" [4]. Thinking Machines' widely cited efficiency result states it as a setup detail: the on-policy run behind its headline AIME'24 number does not start from a base model at all, but from a checkpoint already fine-tuned by off-policy distillation on hundreds of thousands of teacher-generated prompts [5]. The dynamics paper states it as a repair: off-policy cold start on teacher-generated rollouts is one of its two remedies for a failing run, and it works by raising the initial overlap ratio [2].

Three different vocabularies, one dependency. The off-policy stage manufactures the overlap that the on-policy stage then exploits, which is why the dynamics paper can predict the fix that the other two treat as housekeeping.

That reframing is what the cost claim runs into. The vendor's own figure is a comparison between the on-policy continuation and an extrapolation: the off-policy baseline for reaching the same score is estimated by extending a log-linear trend, not measured, and the vendor says so [5]. The reduction also holds only when the off-policy dataset that produced the starting checkpoint is treated as free — given in advance, or amortized across many training runs [5]. And in the experiment itself the teacher was Qwen3-8B while compute was charged as though it were Qwen3-32B [5]. None of that makes the result wrong; it makes it a vendor measurement of the second leg with the first leg zeroed out.

The two cost figures in circulation also measure different terms and are routinely added together. One is prompts-to-target, the vendor's unit: how many further prompts of on-policy training lift the checkpoint's benchmark score, measured against an extrapolated estimate of how many off-policy prompts would do the same [5]. The other is per-step: on-policy training has to run the teacher over the student's own rollouts at every optimization step, and that is the term the 2026 efficiency literature is attacking. Truncating the on-policy objective to a 256-token prefix brings the same student's training cost to 2.4 GPU hours, against 12.2 for ten-step full on-policy distillation — both entries in that paper's limited-budget comparison, which does not report the two at a matched step count [6]. The two terms multiply rather than compete, and among the sources reviewed here, only the second has been measured outside the vendor.

Which is also how on-policy distillation comes to lose outright in a published comparison. In that paper's limited-budget comparison — the settings were chosen because their training FLOP sit in the same order of magnitude — those same 12.2 GPU hours of ten-step full OPD scored 3.5 on AIME-25, while off-policy sequence-level KD at 300 steps cost 3.7 hours and scored 6.9 [6]. The seven-day cap on eight A100s is a separate constraint, and it applied to the 60-step full-OPD run with the 8B student rather than to this comparison [6]. The result indicts the budget the method was given, not the method. Dense token-level supervision is expensive per step, and if a FLOP-matched budget buys ten steps of it against three hundred of a cheaper objective, the cheaper objective may simply get further.

The defensible version, then. On-policy distillation wins where the student's decode-time distribution has drifted away from the teacher's data, at a per-step teacher-inference cost, on top of an off-policy stage you are paying for whether or not the comparison counts it. Among the sources reviewed here, its order-of-magnitude savings claims rest on the vendor's own measurement.

The objective is a sharpness dial inside the overlap — and the heuristic you set it with does not apply

Once the student is in the neighborhood, the objective decides how hard it sharpens. In practice this is one number. In Hugging Face's GKD trainer, `beta` interpolates the generalized Jensen-Shannon divergence: at 0.0 the loss approximates forward KL, at 1.0 reverse KL, and it defaults to 0.5 [12]. A second float, `lmbda`, sets the fraction of on-policy student-generated data, from 0.0 to 1.0 [12]. Two of the four decisions are two public configuration floats.

Practitioners set the first of them using a story: forward KL is mean-seeking and covers the teacher's whole distribution, reverse KL is mode-seeking and commits to the teacher's peaks, so pick reverse KL when you want clean generations. The story is load-bearing in a great deal of published reasoning, and a COLING 2025 analysis argues it does not survive contact with language models.

Its argument is structural. Mean-seeking and mode-seeking behavior are consequences of two assumptions: that the student distribution is unimodal, Gaussian-shaped, and that both distributions are continuous [7]. A softmax over a vocabulary is neither. Strip the assumptions and "neither mode-seeking nor mean-seeking properties manifest in KD for LLMs" [7]. In their demonstration experiments the two objectives converge to the same optimum given enough epochs — past 50 — and the early-epoch behavior is inverted from the folklore: forward KL fits the head of the teacher distribution first, reverse KL the tail [7]. Real distillation runs stop nearer 10 epochs, far inside the regime where the two have not converged. Which means the divergence choice, as practitioners make it, is a choice about transient early-training dynamics rather than about a limiting behavior.

Three positions are live here and this evidence does not settle them. MiniLLM's case for reverse KL is a measured one: on SST2, its reverse-KL student reaches an expected calibration error of 0.099 against 0.191 for standard forward-KL distillation, with the forward-KL student pushing probability into regions the teacher leaves empty [11]. GKD's position is that the optimal divergence is task-dependent and should be tuned [4]. The COLING position is the only one that supplies a mechanism for how the first two can both be right. What would settle it is an epoch-swept forward-versus-reverse comparison at matched tokens on a modern decoder, and nothing here has run one.

What does have a shared currency is the failure mode, and four groups arguing about different decisions run into it from different sides. The currency is entropy — how sharply the student's distribution concentrates.

A 2026 analysis of the reverse-KL gradient finds that its non-target term pushes the target logit upward even after the target probability already matches the teacher's, so confidence keeps climbing past the point of agreement. On the same setup, student prediction confidence measures 0.91 under reverse KL against 0.75 under forward KL — higher, the paper notes, than the teacher's own — while Distinct-2, a diversity metric, falls from 0.755 to 0.600 [8]. The student ends up surer than the model it was copying, and says fewer different things.

Three other groups arrive at the same pathology from different directions. A study decomposing distillation objectives by prefix source and KL direction finds that at 4096-token training lengths reverse KL drives mean per-token predictive entropy close to collapse and pushes responses toward the generation limit; it reports that long-sequence distillation "requires substantial forward-KL weight to prevent entropy collapse" [9]. Entropy-Aware On-Policy Distillation attacks the pathology from the token side, applying forward KL only where the teacher's own distribution is uncertain. On Qwen3 base models at 0.6B, 1.7B and 4B it reports Pass@8 gains of +1.37, +2.39 and +5.05 against standard on-policy baselines [10]. And the dynamics paper finds that aligning prompts to the teacher's template sharpens alignment but causes entropy collapse unless out-of-distribution prompts are mixed in [2].

Four groups, four different questions — which divergence, which sequence length, which tokens, which prompts — and four different corrections: a diversity-aware reverse-KL objective [8], a substantial forward-KL weight at long training lengths [9], forward KL applied only where the teacher is uncertain [10], and out-of-distribution prompt mixing [2]. Taken together, these studies suggest a family of entropy-preserving corrections, often — but not always — implemented by restoring forward-KL-like pressure in uncertain regions. That family is a reading across four papers rather than a position any of them takes. MiniLLM supplies the opposite pole of the same dial, where reverse KL is the fix rather than the failure [11]. The dial has two ends. What none of this supports is the folk rule that told you which end to turn toward.

The same result, measured from outside the optimizer

Over-sharpening is that failure seen from inside the optimizer: too much mass driven into too few places. Which examples you train on is the same choice made from the data side — which regions get sharpened at all. The curation literature makes that choice without ever reading a gradient, and lands in the same place.

s1 curated 1,000 questions with reasoning traces from a pool of 59,029, filtering jointly on quality, difficulty and diversity, and fine-tuned a 32B instruction-tuned model on them in 26 minutes on 16 H100 GPUs [13]. Training on the entire 59,029-example pool, a strict superset of the 1,000, does not offer substantial gains over the 1K selection [13]. The rest of the pool buys essentially nothing.

The ablation inside that result is the part practitioners tend not to carry. It is not the count that does the work. Selecting 1,000 samples at random, or 1,000 maximally diverse ones, or the 1,000 with the longest reasoning traces, each loses around 30% on AIME24 against the jointly filtered selection [13]. Same size, same source pool, roughly a third of the performance gone. Whatever the 1,000 examples are doing, they are doing it by where they land, not by how many they are [13].

LIMO states the mechanism as an explicit hypothesis about the base model rather than about the data. In knowledge-rich foundation models, it argues, sophisticated reasoning emerges through minimal but precisely orchestrated demonstrations — gated on "the latent presence of prerequisite knowledge within the model's parameters" [14]. Its curated set of 800 samples takes AIME24 accuracy to 63.3% and MATH500 to 95.6%, against 6.5% on AIME24 for previous fine-tuned models trained on far more data, with a 45.8% absolute improvement across out-of-distribution benchmarks [14].

And the failure side, from three years earlier, describes the same machine from the other direction. Fine-tuning open models on ChatGPT outputs produced students that crowd workers rated as competitive with ChatGPT, while targeted evaluations found they "close little to none of the gap from the base LM to ChatGPT on tasks that are not heavily supported in the imitation data" [15]. The paper's own summary is that imitation models are "adept at mimicking ChatGPT's style but not its factuality" [15]. That paper does not put it in these terms, and could not have: one reading that fits its result is that style lives in the high-probability region a reweighting account can move, and factual capability does not.

Three measurements, no shared apparatus: a token-level ablation, a data-curation ablation inside one paper, and a human-evaluation study from before any of this vocabulary existed. Binding them together is our move, not theirs — none of the three describes the mechanism the others measure. But one reading fits all three: in these settings, the optimization mostly moves mass the student already has.

Less is more is not a law, though, and the counter-evidence is domain-conditional rather than contradictory. A 2025 preprint studying code-reasoning distillation into small non-reasoning models finds performance non-monotonic in data quantity — a valley where more data initially hurts — reports that easier coding questions beat harder ones in low-data regimes, and finds, surprisingly, that "the correctness of outputs in training data makes no difference to distillation outcomes" [16]. Another preprint reaches better AIME results than s1's 1K using a pool of 90 manually selected reasoning instances plus prompt-based augmentation [17]. Both are unreviewed. Neither overturns the curated-small result; both suggest the enabling condition is the base model's pretraining coverage, which is exactly what LIMO says out loud.

That enabling condition is also where the frame stops holding.

Where reweighting stops

Three things resist this frame.

The first is a clean counterexample. Gemma 2 trained a 2B model twice on identical 500B-token budgets, once from scratch and once distilling from a 7B teacher, and the distilled model averaged 67.7 against 60.3 across three benchmarks [18]. Same data, same tokens, same student architecture. The only difference is the supervision signal, and it is hard to describe that gap as a redistribution of something the student already had.

The second is the dynamics paper's own qualification. Thinking-pattern consistency is not sufficient; a teacher must also offer genuinely new capabilities beyond what the student has already seen, and new knowledge — from further post-training, for instance — enables larger transferable gains even when overlap is already high [2]. The mechanism paper is explicit that its own condition has a second half that its own frame does not explain.

The third closes the boundary from the compute side. Apple's accounting is that distillation beats supervised learning only inside a budget: with many students to train or a teacher that already exists, distillation wins up to a compute level that scales predictably with student size, but "if only one student is to be distilled and a teacher also requires training, supervised learning is generally preferable" [3]. Distillation is an efficiency, not a capability. Given enough tokens, ordinary supervised training gets there anyway.

And then there is the case where the boundary stops being academic. Nothing among the sources reviewed here measures safety behavior as latent in a base model the way mathematical reasoning is latent in a model pretrained on mathematics; on the reweighting account it would then have to be installed rather than drawn out, which is an inference from the frame and not a finding any of the safety papers reports. What those papers do show is that the recipes for restoring it after distillation disagree with each other, and the disagreement is not small.

STAR-1 filters 1,033 samples from an initial pool and reports an average safety improvement of 40% across four benchmarks for a marginal 1.1% average decrease in reasoning across five reasoning tasks [19]. RealSafe-R1 needs 15k safety-aware reasoning traces generated by DeepSeek-R1 itself, kept inside the model's original generation distribution, and evaluates everything at a fixed temperature of 0.6 and top-p of 0.95 [20]. An ACL 2026 study of the same problem concludes the opposite of both: "directly distilling safe responses from DeepSeek-R1 fails to significantly enhance safety", measuring the PAIR attack success rate falling only from 66% to 54% on a distilled 7B student even after filtering the training responses [21].

The damage the recipes are repairing is also not in a consistent direction. Evaluating six DeepSeek-R1-distilled models against their pre-distillation counterparts on a Chinese safety benchmark, four declined and two improved; the worst decline was 9.76% in overall risk-identification accuracy [22]. Compression tells the same story in an older setting: across 18 compression methods and settings, each trained with five random initializations, worst-group accuracy on MultiNLI and SCOTUS degrades for most compressed BERT models while on CivilComments it improves for most of them [23].

So the universal claim — distillation degrades safety — is not what the evidence supports. What it supports is narrower and more awkward to act on: distillation perturbs these properties unpredictably, per teacher-student pair, in a direction you cannot know in advance. Which is only actionable if you can measure the perturbation.

The instrument is looser than the fight

You may not be able to. The effect sizes in that dispute are being arbitrated with an instrument whose own variation is rarely reported beside them.

A 2025 preprint asked how often the same model gives the same verdict on the same prompt. Across four instruction-tuned models, 876 harmful BeaverTails prompts, four temperatures and five random seeds, 18–28% of prompts flipped between refusing and complying [24]. Its reliability headline carries a qualifier that citations tend to drop: single-shot evaluation agrees with the multi-sample ground truth only 92.4% of the time when pooling across temperatures, while at a fixed temperature — how benchmarks actually run — agreement is 94.2%–97.7%, and 97.7% at greedy decoding [24]. The authors still recommend at least three samples per prompt [24]. One unreviewed preprint by a single author, on models that are not the ones in the safety-recipe dispute: a finding, not a settlement.

The observation beside it is documentary. Consider the evaluation behind those per-model safety deltas across six distilled models [22]. Its archived full text records no sampling temperature, no seed count and no samples-per-prompt figure anywhere, and among the sources reviewed here most of this dispute's papers report none of the three. RealSafe-R1 is the exception, fixing decoding at a temperature of 0.6 and top-p of 0.95 and saying so [20]. When one paper reports a 1.1% average reasoning cost [19] and another a 9.76% safety decline [22] and neither says how many samples produced the estimate, the disagreement between them is not yet a disagreement about the world.

A prompt-level disagreement rate is not an error bar on a benchmark delta; the two are different statistics. The narrower claim that survives is that the effects under dispute may fall inside sampling variance nobody has reported. Even at fixed temperature the remedy is cheap — several samples per prompt, a confidence interval around the delta, the protocol printed beside it. Protocol problems are fixable in a way that recipe disputes are not.

What to watch on your own run

Four things follow.

Watch the overlap ratio in the first hundred steps. It is cheap to compute, it separates successful runs from failing ones from the outset, and early mismatch is the one failure mode this evidence says training does not recover from later [2]. If it is stagnant, change the teacher or cold-start the student; do not wait for it to improve.

Budget for the cold-start checkpoint explicitly. Every on-policy result reviewed here begins from one [2, 4, 5], and the efficiency claims that make on-policy attractive count it as free. If you do not already have that dataset, you are paying for the first leg too.

Ablate curation criteria, not sample count. The reproducible finding is that same-sized selections drawn from the same pool differ by roughly a third on hard benchmarks depending on how they were filtered [13]. Sweeping the number of examples measures the least informative axis available.

Take at least three samples per prompt before believing any safety delta, your own included — the number the stability study itself recommends [24]. Even at a fixed temperature, single-shot agreement with the multi-sample ground truth is only 94.2%–97.7% [24], and the deltas being argued over are small enough that nobody has shown they sit outside sampling variance. Put an interval around the delta, and report the protocol — the temperature, the seed count, the samples per prompt — because the field's current inability to compare its own results traces directly to their absence.

The four decisions are real, and the machine underneath them is simpler than the decision tree suggests: teacher, sampling, objective and data are four ways of managing one distance. Managing distance moves what is already there; on the evidence reviewed here it does not install what is not. Where the evidence does show something installed — a teacher carrying genuinely new post-training knowledge, a pretraining budget spent under a teacher's supervision — the gain sits outside the reweighting account rather than inside it. The cases where you need that installation — new capability, safety behavior, anything the base model has no latent purchase on — are exactly the cases where the current measurement apparatus cannot yet tell you whether it worked.