One query recovers most of on-policy distillation's gains

On-policy distillation (OPD) trains a smaller student language model by having it generate its own rollouts and then giving it dense, token-level feedback from a larger teacher model, rather than relying on sparse reward signals. Past work on OPD has mostly studied how the algorithm behaves; a new paper by Zixuan Fu and co-authors instead asks how much training data OPD actually needs, and tests the extreme case: training on a single query.

The result: one-shot OPD, trained on just one query, keeps improving for hundreds of training steps and recovers most, though not all, of the gain that training on a full dataset achieves, and this holds across multiple task domains and model families the paper does not name individually.

The authors explain this through state coverage, the share of the training states that full-data OPD visits which a given query's rollouts also reach. A single query already reaches 71.5% state coverage, most of it within the first 100 training steps. Adding more semantically distinct queries raises coverage and validation accuracy together; at 16 queries, coverage reaches 98.9% and the resulting training matches full-data OPD.

What does not change with more data is the speed of alignment, how quickly the student catches up to the teacher. That pace stays similar whether OPD trains on one query or the whole dataset, and even once the relevant states are covered, absorbing them still takes hundreds of steps. The authors put it directly: "OPD is therefore data-overfed but algorithm-starved." Its rollouts expose broad supervision quickly, they write, but the student absorbs that supervision increasingly slowly.

The same pattern extends to multi-teacher OPD (MOPD): 16 semantically diverse queries per domain match full-data MOPD performance too, a separate result from the single-teacher 16-query case above. As a further stress test, the authors swap in content-light templates and off-domain queries drawn from WildChat; both approach, though do not fully match, the real-query baseline. That leads the authors to note that task content and the state coverage it induces can come apart, that is, a query's subject matter does not necessarily determine how much of the training state space it reaches.

The paper closes by saying it hopes these findings redirect future OPD work toward step efficiency, making the slow absorption phase faster, and prompt a re-examination of what data and mechanisms are really behind OPD's recent successes in frontier post-training.

Key facts

Training on a single query, called one-shot OPD, keeps improving for hundreds of steps and recovers most, not all, of full-data on-policy distillation's gain, across multiple task domains and model families.
That query's rollouts alone reach 71.5% state coverage, the share of states full-data OPD visits, mostly within the first 100 steps; 16 semantically distinct queries push coverage to 98.9% and match full-data training.
Coverage climbs fast, but alignment speed does not: absorbing even a fixed set of states takes hundreds of steps whether OPD trains on one query or the whole dataset, which the authors describe as OPD being data-overfed but algorithm-starved.
The same 16-queries-per-domain result holds for multi-teacher OPD (MOPD); content-light templates and off-domain WildChat queries approach, but do not fully match, the real-query baseline.
The authors say task content and the coverage it induces can come apart, and hope the findings push future work toward OPD's step efficiency and a re-examination of what actually drives its recent gains in frontier post-training.

Why it matters

On-policy distillation has become one of the standard ways to train smaller models on a larger teacher's behavior during LLM post-training, and this paper's headline surprise is that most of its benefit does not need a large, carefully assembled dataset: a single query gets a student most of the way to full-dataset performance. The explanation offered, state coverage, reframes what counts as enough data for OPD: coverage saturates fast, since 16 queries already reach 98.9% of it, so the real bottleneck the authors point to is algorithmic, how slowly a student absorbs supervision once it is exposed to it, rather than how much or how varied the training data is. For teams designing or budgeting distillation pipelines, that shifts the question from how much data is needed to how efficiently the student can learn from what it already sees.

Who it affects

Anyone training smaller student models by distilling a larger teacher's behavior during LLM post-training, especially teams assembling large or highly varied query sets on the assumption that more diverse data means better distillation. It also speaks to researchers working on multi-teacher distillation, where the paper reports the same 16-queries-per-domain pattern, and to anyone building distillation algorithms, since the authors frame the slow, unchanged part, how fast a student absorbs supervision, as the open problem worth attacking next.

How to use it

This is a research paper, not a released product, so there is nothing to install or license. Its practical suggestion for anyone running an OPD pipeline is to check state coverage, the share of full-dataset training states a smaller query set's rollouts actually reach, before assuming a bigger or more varied dataset is the fix. The paper's own numbers suggest a modest set of semantically distinct queries, on the order of 16, can push coverage and accuracy close to full-dataset results, and that even content-light or off-domain queries get partway there. What the paper does not offer is a shortcut for the other bottleneck it identifies: however coverage is reached, the student still needs hundreds of training steps to absorb what the teacher is showing it.

How solid is it

The paper reports its state-coverage explanation holding across multiple task domains and model families, and extends it to a second setting, multi-teacher OPD, plus a stress test that swaps in content-light and off-domain queries, a reasonable spread of checks for one result. Set against that, the text gives no model names and no benchmark names; the one dataset it does name, WildChat, appears only in the off-domain stress test, and it gives no absolute accuracy figure for the full-data baseline itself, only relative comparisons saying other setups recover most of, match, or approach it. Beyond first author Zixuan Fu, the paper names twelve co-authors, though no institution is given, and the work is a preprint posted to arXiv.

Risks and caveats

The single-query result recovers most, not all, of full-data OPD's gain, so it is not shown here as a strict replacement for full-data training. The stress test with content-light templates and off-domain WildChat queries only approaches the real-query baseline, a weaker outcome than the match reported for 16 real queries, so query content is not shown to be entirely irrelevant. And the slower half of the paper's own story, why absorbing supervision takes hundreds of steps regardless of data, is left as an open question for future work rather than something this paper resolves.

“OPD is therefore data-overfed but algorithm-starved.”

— the paper's authors