Glonce

Inverse reasoning model lets Qwen3-0.6B self-distill from answer-only data

researchThe Glonce editorsarXiv🔥58Glonce rating58Impact45Novelty55Relevance100Surprise25Players15Credibility70

The paper starts from a tradeoff in post-training reasoning LLMs. Supervised finetuning typically needs reasoning traces, which is data inefficient because in many applications one only has prompt-answer tuples and no groundtruth chain-of-thought (CoT). Reinforcement learning, for example with verifiable rewards, can work without CoT data, but it is much more compute-inefficient: each optimization step needs multiple autoregressive completions, and none of them is guaranteed to give a non-zero policy gradient.

The authors explore a middle ground: a self-distillation technique that uses answer-only data and treats CoT as latent variables. It relies on training a partner "inverse reasoning" LLM that generates the original model's reasoning traces conditioned on the answer. The inverse model is trained once and reused across incoming batches of answer-only data, and all generation happens offline and up front. The authors say this makes the approach particularly suited to continual post-training.

The experiments use Qwen3-0.6B on the Countdown and GSM8K tasks. After GRPO training has saturated, a single offline distillation round from the inverse model roughly doubles pass@1 on hard prompts. On Countdown, it also outperforms rejection-sampling self-distillation on every hard-tail metric at every mix ratio, with smaller gains on GSM8K. The authors point out that rejection sampling by construction has no signal on problems the model never solves on its own.

Continued GRPO on the fresh data keeps an edge at hard pass@1, but it trails distillation at pass@8 and pass@32 while regenerating rollouts at every step.

The inverse model also turns out to have a second use as an accurate verifier of correctness. When used for best-of-n selection at test time, it raises selection accuracy on Countdown from 0.753 with majority voting to 0.969.

Key facts

  • The method trains an inverse reasoning LLM that generates reasoning traces conditioned on the answer, so a model can be self-distilled from prompt-answer pairs with no ground-truth CoT.
  • With Qwen3-0.6B on Countdown and GSM8K, one offline distillation round after GRPO saturation roughly doubles pass@1 on hard prompts.
  • On Countdown, it beats rejection-sampling self-distillation on every hard-tail metric at every mix ratio, with smaller gains on GSM8K.
  • Continued GRPO on fresh data keeps an edge at hard pass@1 but trails distillation at pass@8 and pass@32.
  • Used for best-of-n selection, the inverse model raises Countdown selection accuracy from 0.753 (majority voting) to 0.969.

Why it matters

Post-training reasoning models forces a choice between two costly options: supervised finetuning that needs reasoning traces most datasets lack, or reinforcement learning that burns compute on many rollouts per step. This paper proposes a middle path that works from prompt-answer pairs alone, treating the chain of thought as a latent variable. It also targets a specific weakness: rejection sampling has no signal on problems the model never solves by itself, and the inverse model is meant to help exactly there.

Who it affects

Teams that post-train reasoning LLMs and hold data in the form of prompts and final answers, without written-out reasoning. The authors say the approach suits continual post-training, where new batches of answer-only data keep arriving. The evidence so far comes from a 0.6B-parameter model, so the near-term audience is researchers studying post-training methods.

How to use it

As described, the workflow has three steps: train the inverse model once, reuse it on each incoming batch of answer-only data to generate reasoning traces offline and up front, then run a distillation round on those traces. In the paper this round came after GRPO training had saturated. The inverse model can also rank candidate answers for best-of-n selection at test time. No code release or dataset availability is mentioned in the source.

How solid is it

The claims come from the paper's abstract and rest on one small model, Qwen3-0.6B, and two tasks, Countdown and GSM8K. The headline numbers are a roughly doubled pass@1 on hard prompts and a Countdown selection accuracy of 0.969 against 0.753 for majority voting. The size of the gain over rejection-sampling self-distillation is not quantified, and no numeric GSM8K results are given beyond 'smaller gains'. The 0.753 to 0.969 figure is for Countdown only. This is an arXiv preprint.

Risks and caveats

No results are reported for models larger than Qwen3-0.6B or for tasks other than Countdown and GSM8K, so it is unknown whether the gains hold at scale or on other problems. Continued GRPO retains an edge at hard pass@1, so distillation does not win on every metric. Gains on GSM8K are smaller than on Countdown. No compute cost, training time or token counts are given, so the efficiency argument cannot be checked against the numbers in the source.

“rejection sampling by construction has no signal on problems the model never solves on its own”

— From the paper's abstract