Negative self-distillation trains LLMs to avoid flawed reasoning

On-Policy Self-Distillation (OPSD) has become a popular way to let large language models improve themselves by acting as their own teacher, using privileged information such as ground-truth solutions to guide training. According to a new paper, however, recent findings show that OPSD can severely degrade performance on complex reasoning tasks: by forcing the student model to imitate an artificially confident reasoning trace conditioned on that privileged information, OPSD suppresses expressions of uncertainty and penalizes the exploratory, self-correcting behavior that difficult problems actually require.

Rongcan Pei and co-authors respond with Negative Self-Distillation (NSD), a framework that flips the direction of training: instead of pulling the model toward a privileged ground-truth answer, NSD pushes it away from a flawed one. The model itself generates a question-specific negative example, essentially playing the role of a "careless reasoner," and training then moves the student's output distribution away from this self-generated negative teacher. None of this relies on ground-truth answers or any other external supervision.

Simply applying an unlearning objective to create that divergence would be risky, though, because tokens that encode flawed reasoning are entangled with ordinary linguistic tokens; penalizing both indiscriminately could badly damage the model's basic language ability. The paper's fix is a dynamic gating mechanism that automatically identifies and isolates the reasoning-critical tokens, so that gradient updates target only the behavioral flaws in reasoning while leaving the model's underlying linguistic priors intact.

According to the authors, NSD consistently outperforms both OPSD and other label-free, self-bootstrapping reinforcement learning baselines in their experiments. The abstract does not give specific accuracy numbers, benchmark names, datasets, or model sizes for these comparisons, nor does it explain in more detail how the gating mechanism tells reasoning-critical tokens apart from basic linguistic ones beyond stating that it does; and no author affiliations or institutions are given in the text itself, only listing metadata that credits Rongcan Pei.

Key facts

OPSD forces models to imitate an artificially confident reasoning trace built on privileged, ground-truth information, which suppresses uncertainty and penalizes the exploratory, self-correcting behavior complex reasoning requires.
Negative Self-Distillation (NSD) has the model generate its own negative example, a self-styled "careless reasoner," and trains the student to diverge from that self-generated flawed trace rather than imitate any ground-truth answer.
NSD needs no ground-truth answers or external supervision at all; both the negative example and the divergence signal come from the model itself.
A dynamic gating mechanism isolates reasoning-critical tokens from ordinary linguistic tokens so only the flawed-reasoning behavior is penalized, avoiding the damage to general language ability that naive unlearning would cause.
The authors report NSD consistently outperforms OPSD and other label-free, self-bootstrapping RL baselines, though the abstract gives no accuracy numbers, benchmarks, or model sizes to quantify the margin.

Why it matters

Most self-improvement pipelines for reasoning models lean on some form of distillation from privileged information, so a known failure mode in the most common version, OPSD, has broad relevance to how reasoning models get trained. The paper's account of imitating confident, privileged reasoning traces as actively suppressing uncertainty and exploration helps explain why some self-distilled models get worse, not better, at hard multi-step problems, and NSD is offered as a way to keep the self-improvement loop going without inheriting that specific failure mode.

Who it affects

The direct audience is researchers and engineers building reasoning-focused language models through self-improvement or reinforcement learning pipelines, in particular anyone currently relying on OPSD or a similar self-distillation setup that conditions on privileged, ground-truth traces. Because the abstract names no specific benchmark, model family, or institution, it is not possible to say from the text alone which existing systems or teams are already affected.

How to use it

NSD is described as a training-time method inside a research paper, not a packaged tool, model checkpoint, or product with its own release. Applying the idea as written would mean having a model generate its own negative, deliberately flawed reasoning trace for a given question, then optimizing the student to diverge from that trace with a gated, unlearning-style objective that only touches reasoning-critical tokens and leaves ordinary language tokens untouched. The text names no benchmark, dataset, task domain, or model size or family that the method was tested on.

How solid is it

The evidence offered is a single empirical claim: the authors say NSD consistently outperforms OPSD and other label-free, self-bootstrapping RL baselines. The abstract supplies no accuracy figures, score deltas, benchmark names, task domains, or model sizes to back that claim, and no author affiliations or institutions are stated in the text, so the result cannot be independently checked from the source alone; as it stands, it reads as a preprint-stage claim awaiting the kind of detail a fuller paper would supply.

Risks and caveats

The central technical step, the dynamic gating mechanism that is supposed to separate reasoning-critical tokens from basic linguistic ones, is asserted rather than explained: the text states that it does this without saying how. Combined with the missing benchmarks and numbers, that leaves open how large the reported improvement actually is, how it was measured, and how well the token-separation gate would generalize beyond whatever setting the authors tested.