Glonce

DeepSeek's censorship doesn't transfer to distilled GPT-OSS, study finds

researchThe Glonce editorsHacker News🔥73Glonce rating73Impact70Novelty75Relevance95Surprise65Players60Credibility80
DeepSeek's censorship doesn't transfer to distilled GPT-OSS, study finds

Researchers built LineageEval, a benchmark for testing whether the political censorship documented in Chinese frontier models transfers into other models trained on their outputs. Distilling from Chinese open models such as DeepSeek is popular for its cost to performance ratio, and some practitioners assume the models' censorship of China-sensitive topics stays behind when only a skill, not the underlying values, is being distilled. Past experiments testing that assumption were confined to small, artificially steered toy setups. This study ran the test in a real production pipeline: DeepSeek V4 Flash, a frontier Chinese model known to refuse or reframe China-sensitive topics (for example, declining to discuss evidence of Uyghur workers in state labor-transfer programs), was used as a teacher to train an American open model, GPT-OSS-120B, purely to improve financial reasoning.

To isolate political censorship from general answer quality, the team built 152 matched prompt pairs (304 prompts total): each sensitive topic was paired with a structurally identical control on a different country, such as the Great Leap Forward against the Holodomor, Tiananmen against Gwangju, the Xinjiang labor-transfer programs against Uzbek cotton, Taiwan against Kosovo, and a Henan village-bank freeze against a Cyprus bail-in. Half the pairs (76) were core political topics and half (76) finance-adjacent economic events with political weight, since the practical question was whether a bank-facing model would whitewash a depositor loss, not whether it would discuss a massacre. Four judge models from four different labs, xAI's Grok 4.20, Google's Gemini 3.5 Flash, OpenAI's GPT-5 Mini and Anthropic's Claude Sonnet 4.6, scored every response 0 to 100 for censorship; the judging setup was validated against 96 human-scored responses, matching them at a Pearson correlation of 0.948 and landing within 10 points of the human score 81.3% of the time. The teacher was served from self-hosted weights through vLLM, not a commercial API, to rule out provider-side moderation layers muddying the result.

On the matched pairs, DeepSeek V4 Flash itself showed a clear censorship gap: its sensitive-topic answers scored 32.02 points more censored than their controls on average, positive on 79% of pairs, a result the researchers describe as roughly seven standard deviations above chance (p < 0.0001). In one worked example, the teacher's answer on the Great Leap Forward famine cited weather and mismanagement, declined to give a death toll, and praised the government's later food-security record, drawing judge scores of 35, 30, 40 and 8 out of 100 and a human score of 20; its answer on the Holodomor named Stalin, forced collectivization and three to five million dead, scoring 72, 90, 90 and 74, with a human score of 75. The student models distilled from that same teacher showed none of this: three separate 120B training arms scored gaps of only 3.94, 3.70 and 2.58 points, positive on just 52%, 55% and 55% of pairs respectively, statistically indistinguishable from the untouched base model. Asked the same Great Leap Forward question, the distilled model named the collectivization, the People's Communes and inflated procurement quotas, scoring 88, 100, 92 and 87. Self-distillation, where the model was trained on its own corrected continuations instead of the teacher's, produced matching results on every random seed.

The financial-reasoning gains that motivated the exercise held up on their own terms. At an 8,000-token generation budget, the released 120B scores 83.61% on FinanceReasoning, ahead of Kimi K3 (81.93%) and Inkling (65.13%), while costing $0.00025939 per query against $0.01605 for Inkling and $0.04141 for Kimi K3, 62 times and 160 times cheaper respectively, and completing 98.7% of problems inside the budget. At an expanded 100,000-token budget the larger models pull back ahead on raw accuracy (Kimi K3 89.92%, Inkling 88.24%, DeepSeek V4 Flash 85.71%), but the researchers argue the tighter budget better reflects real deployment constraints. Because no external model's output entered the self-distilled arm's training signal, that is the version being shipped, as the gpt-oss-20b-finance weights on Hugging Face, alongside LineageEval's prompts, matched controls, judge rubric, evaluation code and models. The 120B serves from a single H100 or A100 at roughly 63GB in native MXFP4 with an 80MB adapter; a smaller 20B version needed its expert layers adapted, not just attention, to recover its gains, reaching 74.79% against a 64.71% base at 23% lower cost per query. A robustness check re-running the audit at the MXFP4 precision the model actually serves at reproduced the base and teacher-distilled results, and the self-distilled arm's censorship gap came back at +0.26 points at both precisions, identical to the hundredth (95% CI on the difference [-2.52, +3.02]), though its truncation count on the token budget rose from 46 to 55 of 152 responses.

Key facts

  • DeepSeek V4 Flash, used as teacher, showed a matched-pair censorship gap of +32.02 points, positive on 79% of 152 pairs (p < 0.0001); the GPT-OSS-120B students distilled from it scored only +3.94, +3.70 and +2.58, positive on 52%, 55% and 55% of pairs, matching the untouched base model.
  • LineageEval pairs 76 core-political topics (Tiananmen, Taiwan, Xinjiang labor transfer) with 76 structurally matched finance-adjacent ones (Henan bank freeze, Cyprus bail-in), scored by four judge models from four labs and validated against 96 human scores at a Pearson r of 0.948.
  • On the Great Leap Forward famine, the teacher's answer scored 35, 30, 40, 8 out of 100 (a whitewash) versus 72, 90, 90, 74 for the matched Holodomor answer; the distilled student's Great Leap Forward answer scored 88, 100, 92, 87.
  • The shipped model reaches 83.61% on FinanceReasoning at an 8,000-token budget, beating Kimi K3 (81.93%) and Inkling (65.13%) at $0.00025939 per query, 62 and 160 times cheaper respectively; at a 100,000-token budget the larger models overtake it (Kimi K3 89.92%, Inkling 88.24%, DeepSeek V4 Flash 85.71%).
  • The team released the gpt-oss-20b-finance weights on Hugging Face plus LineageEval's prompts, judge rubric, evaluation code and models; a re-run of the audit at MXFP4 serving precision reproduced the results, though the self-distilled arm's truncation count rose from 46 to 55 of 152 responses.

Why it matters

Distilling from Chinese open models like DeepSeek is attractive for cost and performance, and one common argument for it is that a model can absorb a narrow skill, such as financial reasoning, without absorbing the teacher's political censorship. That belief had only been tested in small, artificially steered setups. This study runs the test in a real production pipeline with a genuine frontier Chinese teacher, and finds the belief holds: political censorship measured on 152 matched topic pairs did not detectably transfer from DeepSeek V4 Flash into the distilled GPT-OSS-120B, even though the same teacher shows a clear, statistically significant censorship gap on its own answers. The result speaks directly to a live policy argument, which the researchers say is being argued by Washington and corporate procurement desks from opinion rather than evidence, over whether building on Chinese open weights imports the teacher's politics.

Who it affects

Teams and companies choosing between Chinese and American open models for cost to performance reasons, particularly in finance, where a bank's practical concern is whether a model whitewashes a depositor loss rather than whether it discusses a massacre. Policymakers and procurement bodies weighing restrictions on Chinese open weight models gain a data point instead of an opinion. AI safety researchers studying subliminal learning, the transmission of traits through unrelated training data, get a case where the channel appears not to fire when teacher and student share no initialization.

How to use it

The gpt-oss-20b-finance weights are open on Hugging Face, with a playground to try the model directly. The full LineageEval instrument, its 304 prompts and 152 matched pairs, the judge rubric and the evaluation code are on GitHub, along with the trained models. The shipped 120B runs on a single H100 or A100 GPU (about 63GB in native MXFP4 plus an 80MB adapter), with two GPUs recommended for high-concurrency production; per-query cost is about $0.00026 at the 8,000-token budget the team recommends for scoped finance tasks. The smaller 20B needs its expert layers adapted, not just attention, to reach its reported gains.

How solid is it

The audit is large by the field's standards: 152 matched prompt pairs split evenly between core-political and finance-adjacent topics, scored by four judge models from four independent labs (xAI, Google, OpenAI, Anthropic) and cross-checked against 96 human scores (Pearson r 0.948, mean absolute error 6.08 points, within 10 points of the human score 81.3% of the time). The teacher was served from self-hosted weights via vLLM specifically to exclude commercial-API moderation layers as a confound. The researchers confirm zero China-sensitive content anywhere in the training data itself (220 prompts, 176 on-policy examples, 181 retained completions, 1,574 source problems), which is what makes this a clean test of the subliminal channel rather than a test of leaked content. A robustness check re-ran the full core-political audit at the MXFP4 precision the model actually serves at; the base and teacher-distilled arms reproduced, and the self-distilled arm's gap came back at +0.26 points at both precisions, identical to the hundredth (95% CI on the difference [-2.52, +3.02]), with 96% classification agreement, though its truncation count rose from 46 to 55 of 152 responses, past the study's own five-point tolerance.

Risks and caveats

The finding covers one teacher (DeepSeek V4 Flash), one skill domain (financial reasoning) and one student architecture (GPT-OSS); it is not a general claim that distillation from Chinese models never carries politics, only that it did not in this pipeline. The self-distilled arm's rising truncation count at serving precision is flagged by the researchers themselves as exceeding their own tolerance, a caveat they report rather than smooth over. The work is self-published, a Show HN post on the researchers' own site, rather than peer-reviewed, and the favorable cost and performance numbers for the shipped model come from the same team that ran the censorship audit.

“Political censorship did not transfer.”

— the researchers' report