Distillation Copies the Transcript, Not the Teacher

What a Student Actually Receives

Meditron-7B, the open medical model from EPFL, ships with a model card that says: "We strongly recommend against using this model in production for natural language generation or for professional purposes related to health and medicine" [2]. Someone took it at its word in the worst possible way.

The recipe was ordinary. Issue 48,000 instruction queries to Meditron-7B; keep the 25,000 benign instruction-response pairs that come back; use them as training data for a cheap, parameter-efficient fine-tune of an off-the-shelf LLaMA-3 8B [1]. No model weights, no safety filters, no training data, no gradients — output-level access only, at a cost of about $12 [1]. The surrogate matched its teacher on benign medical inputs. It also produced unsafe completions for 86% of adversarial prompts, against 66% for Meditron-7B itself and 46% for the untuned base checkpoint it started from [1].

Nothing in those 25,000 exchanges was harmful [1]. The attacker never asked for anything dangerous. The copy came out more dangerous than either model that went into it.

A teacher is the model you query. A student is the model you train on what the teacher said. Black-box or behavioural distillation means you never see inside the teacher — you see only its outputs, the way any API customer does. The training mixture is the set of prompts and responses you actually keep. The student learns the mixture, not the teacher.

The position paper that names this most cleanly is Wenshuo Wang's, which argues that the field should stop treating a retained task score as evidence of a retained capability. Reframe distillation as a lossy projection onto whatever observables you sampled, and the flaw in that inference becomes visible immediately.

"Matching a projection is not the same as matching the full capability vector." [3]

— Wenshuo Wang, Knowledge Distillation Must Account for What It Loses

Wang's taxonomy of what falls outside the projection is not speculative: predictive distributions, internal representations, robustness, calibration, subgroup behaviour, privacy and memorization, reasoning faithfulness, on-policy stability, refusal boundaries, grounding, synthetic-data diversity. He calls these off-metric losses — concrete, recurring, measurable, and almost never reported, because studies report what students retain rather than what they lose [3].

Which means the interesting question is never whether a student is a good copy. It is which observables were in the sample.

When the Projection Lines Up With the Metric

In the best case the projection loses nothing anyone is measuring, which is why the technique is worth using at all.

When DeepSeek released R1 in early 2025, it also released a family of students fine-tuned on samples curated with R1, built on off-the-shelf Qwen2.5-Math, Qwen2.5 and Llama bases [4]. The results were not marginal. DeepSeek-R1-Distill-Qwen-32B reaches 72.6 pass@1 — problems solved on the first attempt — on AIME 2024 and 94.3 on MATH-500, against R1's own 79.8 and 97.3 [4]. The student lands a few points under its teacher on both. DeepSeek's card states the comparison that mattered commercially: "DeepSeek-R1-Distill-Qwen-32B outperforms OpenAI-o1-mini across various benchmarks, achieving new state-of-the-art results for dense models" [4].

The economics are the other half. A benchmarking study of distilled language models puts the cost of creating a distilled 8B model at less than 0.1% of the compute budget of training a vanilla 8B model from scratch [5]. That is not compression. That is a different production function.

And it works for exactly the reason the projection model predicts. You sampled long reasoning traces. Reasoning traces are therefore what came across. Reasoning benchmarks are what you measured. The observable you sampled and the observable you scored are the same observable, so the transfer looks near-total.

Narrow the question by one degree and it stops looking total. The same position paper points to metamorphic testing of distilled code models, where students preserve conventional accuracy while failing to mimic the teacher under behaviour-preserving transformations of the input [3]. They passed the metric and failed the behaviour the metric was standing in for.

If the projection carries only what you sampled, what happens to a behaviour you never once asked the teacher to perform?

Refusal Is the Behaviour Nobody Asks For

Nobody queries a teacher model to collect refusals. Refusals are the failed requests, the ones you discard. A corpus of 25,000 helpful medical answers [1] was never assembled to demonstrate declining — so declining gets little or no gradient, and the student learns to be helpful about everything.

The academic measurements scale that past the anecdote. In the SafeChain evaluation of large reasoning models, only 32.3% of responses by R1-70B were judged safe — the authors note this implies that fine-tuning with long chain-of-thought does not by itself enhance safety [7]. R1's 7B student was measured on two harmful-prompt benchmarks and came out in the same range. Safe@1 — the share of responses judged safe on a single attempt — sat at 36.4% on StrongReject and 49.6% on WildJailbreak [6]. On neither benchmark did the student clear half.

There is a widely circulated number in this neighbourhood that deserves calibration rather than amplification. A Cisco security blog post, reporting work by the Robust Intelligence team with University of Pennsylvania researchers, recorded a 100% attack success rate against DeepSeek R1 — every one of 50 randomly selected prompts from HarmBench, a standard harmful-behaviour set, got an affirmative answer [11]. The result was relayed widely enough to appear in the academic safety literature as the framing for further study [12]. But it is a blog-tier report, single-shot, on 50 prompts, and it is about R1 itself rather than its students. And single-shot is doing real work in that sentence. One study tested four instruction-tuned models across 876 harmful prompts and 20 sampling configurations, and found that 18–28% of prompts flip — refused under some seeds and temperatures, complied with under others [10]. A single measurement of refusal is a measurement of one draw.

Why refusal specifically, rather than any other behaviour? Because refusal is a thin object. Arditi and colleagues found that refusal is mediated by a one-dimensional subspace across 13 popular open-source chat models up to 72B parameters: erase that direction from the residual stream — the running vector representation a transformer passes forward through its layers — and the model stops refusing harmful instructions; add it, and the model starts refusing harmless ones. They implemented the jailbreak as an interpretable rank-one weight edit [8]. That finding is the basis for the entire genre of abliterated models on Hugging Face: open checkpoints whose refusal direction has been orthogonalized away, no retraining required [13].

Later work complicates the picture without rescuing it. Across eleven categories of refusal and non-compliance, refusal behaviours turn out to correspond to geometrically distinct directions — yet steering along any one of them produces nearly identical refusal-to-over-refusal trade-offs, acting as a shared one-dimensional control knob. The primary difference between directions is not whether the model refuses but how [9]. Eleven handles, one lever — and a thin thing is easy to fail to copy.

It is also cheap to put back. Fine-tuning R1-7B on SafeChain, a safety dataset written in the same long chain-of-thought style the model already produces, moves both safety scores a long way and leaves the maths score roughly where it was [6].

Safety loss in distillation is a data decision, not a law of the method.

So Why Did the Censorship Not Come Along

Hold that mechanism — refusal is a policy carried in the data — and the loudest fear about distilling a censored model becomes a question someone can instrument. The evidence on it is asymmetric, and the asymmetry is worth naming before the numbers arrive. The claim that censorship does travel — that DeepSeek's own R1-distilled Qwen and Llama students carry R1's censorship — rests on a personal blog write-up rather than on a published measurement, which is not weight enough to carry it. The measurements that do exist run the other way.

On 29 July 2026, the control-tooling startup CTGT published a distillation study built to test a specific fear: that training on a censored Chinese model transmits its censorship. They distilled DeepSeek V4 Flash into GPT-OSS-120B for financial reasoning, and measured with a matched-pair instrument — China-sensitive prompts paired against structurally similar controls, 152 pairs in all, each response scored 0 to 100 for censorship by four independent judge models from four different American labs [15].

A positive gap means the model is more evasive on the sensitive prompt than on its matched control; a gap near zero means the behaviour is not there. The teacher censored. On 76 core-political pairs, DeepSeek V4 Flash scored a matched censorship gap of +45.45, with 87% of pairs showing a positive gap [14]. Every student landed within a point of its unmodified base: the Flash-taught model measured −1.39, indicating no selective avoidance of China-related content [14]. Concretely: the teacher refused the Xinjiang question while answering its Uzbekistan control; the Flash-taught GPT-OSS model answered both [15].

The reason is in the mixture, and CTGT built it that way on purpose. The distillation data consisted of 220 training prompts and their derived examples, containing zero China-sensitive content [14].

"Testing whether censorship transmits through unrelated data requires that it never appear in the data." [14]

— Johnny Yu, Siddarth Mamidanna and Cyril Gorlla, CTGT

Corroboration that this behaviour lives in the mixture rather than the weights comes from the other direction. Perplexity's R1-1776 removed R1's censorship by post-training on roughly 300 identified censored topics and a 40,000-prompt multilingual dataset — Perplexity's own published figures, reported second-hand [17] — reportedly without degrading the model's basic reasoning ability [18]. A behaviour you can subtract by editing data is a behaviour that lives in data. R1dacted shows what that censorship is. Across a curated dataset spanning 96 categories, 97.3% of R1's censorship cases were plain refusal to answer, with 66 of the 96 categories censored 100% of the time [16]. It is the same machinery as ordinary refusal, conditioned on topic.

Two limits bound what that shows. First, scope: this is one model pair, one task family, one distillation method, and CTGT makes no claim that the result applies to safety training, security behaviour, or general refusal policies [15]. Second, standing: CTGT is a commercial vendor of model-control tooling publishing research about model control. What partly offsets that is the instrument — the prompts, the matched controls, the judge rubric and the evaluation harness were released, and the study was independently covered [15]. The claim is checkable by someone who does not trust the claimant, which is the property that matters.

The null result also no longer stands alone. R1dacted put the six models DeepSeek distilled from R1 through the same censorship prompt set — every prompt in it curated because it reliably triggers censorship in R1 — and found overall censorship rates of 0.15% to 0.30%, the two common types appearing in only a very limited number of samples [16]. What survives is a residual form: some reasoning tokens, not always coherent ones, and then a refusal, most of those cases on questions about the Tiananmen Square massacre [16]. The authors read that residue the way the mixture account would, and stop short of concluding it: it is plausible that some censored samples were included in the distillation dataset, they write, and a definitive answer is left to future work [16]. Academics with nothing to sell, measuring DeepSeek's own students, land where the vendor landed.

Both readings can hold, at very different sizes: a mixture with none of the topic in it transmitted none, and a mixture that plausibly carried a few censored samples left a fraction of a percent behind. CTGT's own explanation for its null result points somewhere else entirely. Teacher and student did not share an initialization. That is not a rival account of the mixture — it names a second route into a student, one that runs outside the transcript altogether.

The One Channel That Is Not in the Transcript

Initialization ought to be irrelevant here. Everything so far assumes the student learns what the transcript demonstrates, and a transcript does not carry the weights that produced it. There is one documented channel where that assumption fails.

Take a model, prompt it to love owls, and have it emit nothing but number sequences. Fine-tune a second model on those numbers. The student picks up a preference nobody showed it: in the Nature paper reporting this, the frequency with which students named their teacher's preferred animal rose from 12% to more than 60% for "owl" [19].

The same route carries misalignment. A teacher fine-tuned on an insecure-code corpus transmitted broadly misaligned tendencies through number sequences even after 34 integers with strong cultural or emotional associations — 666, 911, 187 among them — were stripped from the data [19]. Nor is the effect confined to numbers. It appears with code and with chain-of-thought transcripts, and in the code experiments the filter that removed any response containing the target word was discarding only between 2.5% and 5.4% of responses [20]. The filter, in other words, had almost nothing to catch. Prompted classifiers, in-context probes and human inspection all failed to find the trait in the data [21].

The signal here is not semantic, it is geometric. One line of follow-up work localizes the effect to a single steering vector, showing that a student trained on the outputs of a steered teacher learns to imitate that steering — and that system prompts not well approximated by a steering vector are not subliminally learned [22]. Non-semantic data can carry a vector with semantic effects.

That framing predicts the boundary, and the boundary is the reason CTGT's null result is not a refutation. Transmission is reliable when teacher and student share the same model initialization. Students trained on outputs from mismatched base models do not adopt the trait; GPT-4.1 and GPT-4o, which transmit to each other, are the reported exception [19]. The authors also prove a theoretical result — a single sufficiently small gradient step on any teacher-generated output moves the student toward the teacher, given shared initialization — and replicate the phenomenon in an MNIST classifier trained without class logits or handwritten digits [21].

The consequence for anyone who was relying on filtering: filtering is not a guarantee. Not because the filters were sloppy, but because you cannot prove the absence of a trait you have not yet articulated.

The Channel Is Real; Its Boundaries Are Not Settled

The sharpest challenge argues that subliminal learning is a LoRA artifact. LoRA — low-rank adaptation — is the cheap fine-tuning method that updates a small number of added parameters instead of all of them; full fine-tuning updates everything. That paper reports three things: transmission tracks LoRA rank in an inverted U, it disappears under full fine-tuning, and it depends heavily on the context seen during fine-tuning and evaluation. Its conclusion is that the channel is a fragile artifact of hyperparameters rather than a stable route for behavioural transmission [23]. The Nature experiments ran through a commercial fine-tuning API, which does not settle what was happening underneath.

Pulling the other way: the MNIST replication and the gradient-descent theorem are arguments for generality, not for a LoRA quirk [21]. And a study of agent distillation reports a destructive-action bias transferring from a Llama 8B teacher to a Qwen 7B student — a 100% deletion rate in the student against a 20% baseline for that same Qwen model — which cuts directly against the shared-initialization boundary [24].

The phenomenon is real and published in Nature. Which fine-tuning regimes it survives, and which teacher-student pairs it crosses, are open questions right now. Policy is already being written on top of them.

The Machine, and the Argument Being Had About It

That policy is not being argued in these terms at all.

On 23 February 2026, Anthropic published a report identifying what it called industrial-scale campaigns by three laboratories — DeepSeek, Moonshot and MiniMax — to extract Claude's capabilities, through roughly 24,000 fraudulent accounts generating more than 16 million exchanges [25]. Its stated concern was not only commercial: "Illicitly distilled models lack necessary safeguards, creating significant national security risks" [25].

Eleven days earlier, OpenAI's 12 February 2026 memo to the US House Select Committee had made the free-riding case, reporting accounts associated with DeepSeek employees "developing methods to circumvent OpenAI's access restrictions and access models through obfuscated third-party routers and other ways that mask their source" [26]. And in April, testifying in his own suit against OpenAI, Elon Musk conceded that xAI had partly used distillation from OpenAI's models, offering the industry defence: "It is standard practice to use other AIs to validate your AI" [27].

Read through the machine, the safeguard claim is right about the outcome and wrong about the mechanism. The Meditron surrogate was not stripped of its teacher's guardrails; it never received them, because a set of 25,000 benign question-and-answer pairs is not where guardrails get demonstrated [1]. A student does not lose safety. It fails to be given safety.

That distinction is not pedantic, because it determines which interventions can work. If the loss were extraction, restricting access would reduce it. If the loss is a property of what benign transcripts contain, then no access regime produces a safe student — and the fixes that do work are data fixes, available to anyone. SafeChain is one [6]. RealSafe-R1 is another: a dataset of 15k safety-aware reasoning trajectories generated by DeepSeek-R1 under explicit instructions for refusal behaviour, which its authors report improves safety while preserving reasoning [28]. Both are published, both are cheap, and neither requires the teacher's cooperation.

Two defensive techniques now being built point the same way, and inherit the same gap. Antidistillation sampling modifies a model's next-token distribution to poison its reasoning traces for downstream training while preserving practical utility [30]. Antidistillation fingerprinting, accepted at ICML 2026, samples tokens chosen to maximize the fingerprint's detectability in a student after fine-tuning, even when the student's architecture is unknown [29]. These are genuinely clever, and they answer a genuinely different question. They establish where a transcript came from, not what the transcript failed to demonstrate.

Which leaves the exposure the mechanism actually predicts, and it is not the one being legislated. By one newsletter's read of the industry — an assessment rather than a measurement — most distillation in 2026 is endogenous: laboratories training new models on synthetic data from their own prior models [31]. That is precisely the configuration in which the shared-initialization condition is satisfied by construction, generation after generation.

"The signals don't require a foreign adversary to appear. They appear because distillation pipelines transmit them, and every frontier lab uses distillation." [31]

— Turing Post

If that holds — and its scope is exactly what the LoRA dispute leaves unsettled — then the auditing regime that inspects training data is checking the wrong artifact. The shift the literature keeps circling is from inspecting data to auditing models, and from clean-dataset claims to lineage attestation [31]. It is a shift being proposed on top of a mechanism whose boundaries are still being argued about, which is an uncomfortable place to write policy from, and also the only place available.