Glonce

RL paper splits training-inference mismatch in two, proposes DLC fix tested on Qwen3

researchThe Glonce editorsHugging Face🔥54Glonce rating54Impact35Novelty60Relevance95Surprise25Players8Credibility72
RL paper splits training-inference mismatch in two, proposes DLC fix tested on Qwen3

On-policy reinforcement learning for large language models assumes that training responses come from the current policy. In practice an inference engine samples the responses and a separate trainer computes the gradients. The training-inference mismatch between the two makes the training data off-policy. The paper says the mismatch has two sources: the numerics of the engine, and the stale weights of asynchronous sampling. A correction that stabilizes training under one source can fail under the other, so the paper asks which property of a mismatch source decides the correction it needs.

The analysis starts from a split. Evaluating the trainer at the sampler's weights divides each sampled-token error exactly into two parts: a numerical discrepancy of the engine and a policy lag of the trainer. The paper follows both through training and finds they behave differently.

The numerical discrepancy stays approximately constant, so it biases the updates by a nearly fixed amount. Centering the score at each prefix removes its leading bias.

The policy lag is different because the trainer's own updates produce it. The bias the lag leaves enters the next update, which enlarges the next lag and closes a feedback loop. Importance weighting cuts this loop only while the lag ratio stays concentrated. On Qwen3-1.7B it fails at the step where the effective sample size of the ratio falls below one half.

From this account the paper derives the discrepancy-lag correction (DLC). It does three things: it centers the numerical discrepancy, it clips the lag ratio against the sampler's weights, and it resynchronizes the sampler when the effective sample size falls below one half.

The one quantitative result in the abstract is a bound. Under a stale sampler, DLC keeps the KL divergence from on-policy training within twice the KL between two independent on-policy trainings, on Qwen3-0.6B and Qwen3-1.7B. The paper also says DLC's resynchronization prevents collapses that per-token corrections leave in place and that a KL measured on the sampler's prefixes does not detect.

Key facts

  • The paper splits each sampled-token error exactly into a numerical discrepancy of the inference engine and a policy lag of the trainer, found by evaluating the trainer at the sampler's weights.
  • The numerical discrepancy stays approximately constant and biases updates by a nearly fixed amount; centering the score at each prefix removes its leading bias.
  • The policy lag feeds itself: the bias it leaves enters the next update and enlarges the next lag. Importance weighting breaks this loop only while the lag ratio stays concentrated, and on Qwen3-1.7B it fails when the effective sample size of the ratio drops below one half.
  • The proposed DLC centers the numerical discrepancy, clips the lag ratio against the sampler's weights, and resynchronizes the sampler when the effective sample size falls below one half.
  • Under a stale sampler, DLC keeps KL divergence from on-policy training within twice the KL between two independent on-policy trainings, on Qwen3-0.6B and Qwen3-1.7B.

Why it matters

Many LLM reinforcement learning setups sample with one inference engine and train with another system, so the data is never quite on-policy. The paper's claim is that the two sources of this gap, engine numerics and stale weights from asynchronous sampling, need different fixes, and that a fix built for one can fail under the other. Separating them gives a clearer picture of why some training runs collapse, and the paper ties one failure to a measurable signal: the effective sample size of the importance ratio dropping below one half.

Who it affects

Mainly people who run on-policy reinforcement learning for language models with a separate inference engine and trainer, especially with asynchronous sampling where the sampler's weights lag behind the trainer's. The reported experiments use Qwen3-0.6B and Qwen3-1.7B.

How to use it

The practical recipe is DLC: center the numerical discrepancy at each prefix, clip the lag ratio against the sampler's weights, and resynchronize the sampler once the effective sample size of the ratio falls below one half. The abstract does not mention a code release, so the method would have to be reimplemented from the description.

How solid is it

The evidence in the abstract is narrow. The quantitative result is a KL bound of twice the KL between two independent on-policy trainings, on two small models, Qwen3-0.6B and Qwen3-1.7B. The importance-weighting failure is reported on Qwen3-1.7B. No task benchmarks, accuracy or reward figures are reported in the abstract. This is an abstract on Hugging Face Papers, not a full read of the paper's methods and results.

Risks and caveats

Only Qwen3-0.6B and Qwen3-1.7B are named as tested models; nothing is said about larger models. The abstract also does not mention compute cost, training-speed overhead of DLC, or how often resynchronization is triggered in practice. The claim that resynchronization prevents collapses missed by a KL measured on the sampler's prefixes is the paper's own, and the abstract gives no figures for it.

“a correction that stabilizes training under one source can fail under the other”

— Paper abstract, Hugging Face Papers