Glonce

TTPO raises Qwen3-1.7B accuracy without labeled data

researchThe Glonce editorsHugging Face🔥55Glonce rating55Impact65Novelty72Relevance95Surprise45Players25Credibility80
TTPO raises Qwen3-1.7B accuracy without labeled data

A new paper by Aozhe Wang and co-authors tackles a specific gap in how large language models get better at math reasoning. The post-training methods that have recently driven progress, including reinforcement learning (RL) and On-Policy Self-Distillation (OPSD), depend on a ground-truth answer key. That dependency rules them out for test-time training (TTT): continuing to train a model using only what is available while it is actually being used, with no prepared answer key on hand. A natural workaround is to replace the missing ground truth with a majority-vote pseudo-label: have the model generate several answers to the same problem and treat whichever answer comes up most often as a stand-in for the correct one.

The authors say that workaround is fragile on its own. When the majority vote itself happens to be wrong, it corrupts the teacher signal and misleads every token the model is trained on. Their key observation is that the failure is asymmetric: a rollout, one of the model's own sampled attempts at an answer, that disagrees with the pseudo-label tends to be wrong regardless of whether the pseudo-label itself is correct.

Building on that asymmetry, they propose Test-Time Policy Optimization (TTPO). Rollouts that agree with the pseudo-label are trained on further through OPSD; rollouts that disagree are instead penalized through Grouped RL. A token-level selection step refines both branches: on the distillation side, positions the model has already converged on are down-weighted so training does not keep pushing on things it already gets right; on the RL side, only confident errors are penalized, sparing tokens where the model is merely uncertain. The authors say both updates stay well-grounded even when the pseudo-label is wrong frequently, and that majority-vote routing supplies tighter self-supervision as the model improves during training.

Tested with no labels at all, TTPO matches the performance of label-supervised OPSD on five competition-level benchmarks, whose names the paper does not give. On Qwen3-1.7B, a 1.7-billion-parameter model, it raises test-time-training accuracy from 38.0% to 45.2%. The paper also reports a further gain of +25.2% to +36.4% in a 'without thinking' (non-reasoning) setting, without stating which model or benchmark that range was measured on, and says TTPO shows strong generalization across tasks.

Key facts

  • TTPO is a label-free test-time training (TTT) method that replaces ground-truth labels with majority-vote pseudo-labels for math reasoning in large language models.
  • It applies an asymmetric update: rollouts that agree with the pseudo-label are trained on through On-Policy Self-Distillation (OPSD), while rollouts that disagree are penalized through Grouped RL.
  • A token-level selection step down-weights positions the model has already converged on during distillation, and penalizes only confident errors during the RL step.
  • Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks and raises Qwen3-1.7B's test-time-training accuracy from 38.0% to 45.2%.
  • It also yields a further gain of +25.2% to +36.4% in a 'without thinking' (non-reasoning) setting, without a stated model or benchmark, and shows strong cross-task generalization.

Why it matters

The post-training methods that have recently driven progress in math reasoning, including RL and OPSD, need a ground-truth answer key, which is exactly what is missing once a model is already deployed and in use. TTPO's contribution is a way to keep improving the model through that gap: swap the missing ground truth for a majority-vote pseudo-label the model generates itself, but do it in a way that survives the vote being wrong often, by treating rollouts that agree with the vote differently from rollouts that disagree rather than trusting the vote outright. The authors report that this keeps the training signal grounded even when the pseudo-label errs frequently, and that the self-supervision gets tighter as the model improves. If that holds outside this one paper's own five competition-level benchmarks, it points toward models that keep sharpening on the reasoning problems they actually meet in use, rather than only on a labeled set fixed in advance.

Who it affects

The direct audience is researchers and engineers building math and reasoning-focused language models, plus anyone evaluating test-time training as an alternative to conventional labeled post-training. The paper's reported results center on Qwen3-1.7B, a 1.7-billion-parameter model, so the teams most directly represented in what it actually demonstrates are the ones working with lightweight or resource-constrained reasoning models rather than the largest frontier systems.

How to use it

The paper frames TTPO as a procedure carried out during training. For each problem, the model samples several attempted answers, called rollouts, takes whichever answer comes up most often as a stand-in label, then updates itself on the attempts that agree with that label through OPSD and penalizes the attempts that disagree through Grouped RL. Within each of those two updates, a token-level weighting step treats positions the model already handles well, and tokens where it is merely uncertain, differently from tokens where it is confidently wrong.

How solid is it

The central result is that TTPO, using no labels at all, matches label-supervised OPSD on five competition-level benchmarks and separately raises Qwen3-1.7B's test-time-training accuracy from 38.0% to 45.2%, with the authors also reporting strong generalization across tasks. That evidence comes entirely from the paper's own reported experiments. The text does not name the five benchmarks, which limits how directly the comparison against label-supervised OPSD can be checked, and the reported numbers all come from a single 1.7-billion-parameter model.

Risks and caveats

Two things the paper leaves open are worth flagging. The extra gain of +25.2% to +36.4% in the 'without thinking' setting is not tied to a stated model or benchmark, so it is unclear whether it describes the same Qwen3-1.7B setup as the 38.0%-to-45.2% result or a different one. And the text does not say whether TTPO was tried on any model larger than Qwen3-1.7B, so how it behaves at bigger scale is untested by this paper's own account. The mechanism the authors credit for TTPO's robustness, treating agreeing and disagreeing rollouts asymmetrically so a wrong pseudo-label does not corrupt the whole update, is what the reported numbers rest on.