Study finds RL avoids the multi-task conflicts that plague SFT, proposes Parallel-RL
A paper analyzing multi-task reasoning training for large language models reports that Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) behave in fundamentally different ways once several tasks are trained together. In preliminary experiments, the authors observed that SFT suffers severe task conflicts under multi-stage training: as more tasks are added, performance on earlier tasks degrades. RL, by contrast, lets diverse tasks coexist stably during training without that degradation.
The authors trace the difference to the parameter level. Under RL, updates to the model's parameters are sparse and approximately orthogonal across tasks, meaning different tasks tend to adjust different, largely non-overlapping directions in parameter space rather than fighting over the same weights. They back this observation with a theoretical analysis of multi-task gradient interference, the mechanism by which training signals from different tasks can cancel out or distort one another. The analysis draws a distinction between the two training methods: interference under SFT is norm-limited, meaning it scales with the absolute size of the gradient, whereas interference under RL is variance-limited, bounded instead by the gradient variance produced by advantage normalization and on-policy optimization. Because that variance bound is small, RL's optimization directions across tasks end up close to orthogonal, which is what keeps tasks from conflicting.
Building on this finding, the authors propose Parallel-RL, a training paradigm that decouples multi-task training. They state that it significantly improves efficiency and flexibility, though the text does not give benchmark numbers, model sizes, datasets, or an experimental setup to quantify that improvement, nor does it name a publication venue or date beyond the paper itself.
Key facts
- Preliminary experiments show SFT suffers severe task conflicts under multi-stage, multi-task training, while RL enables stable coexistence across diverse tasks
- Parameter-level analysis finds RL induces sparse, approximately orthogonal updates across tasks
- Interference under SFT is norm-limited, scaling with the absolute gradient magnitude; interference under RL is variance-limited, bounded by the gradient variance from advantage normalization and on-policy optimization
- This small variance bound under RL yields near-orthogonal optimization directions across tasks
- The authors propose Parallel-RL, a paradigm that decouples multi-task training and, they say, significantly improves efficiency and flexibility
Why it matters
Multi-task training for LLM reasoning is often treated as roughly interchangeable between fine-tuning methods, but this analysis argues the two behave very differently once several tasks share a training run. SFT accumulates conflicts as tasks are added in stages, degrading earlier tasks as later ones are trained; RL's updates stay close to orthogonal across tasks, so tasks stop competing for the same parameters. If the pattern holds more broadly, it reframes how a model that needs to handle many reasoning skills or domains at once should be trained, favoring RL-style optimization over sequential SFT for that purpose.
Who it affects
Researchers and engineers building multi-task or multi-domain LLM reasoning systems, especially teams currently relying on staged or sequential SFT across tasks and evaluating whether to move toward RL-based training instead.
How to use it
The paper's practical contribution is Parallel-RL, a paradigm that decouples multi-task training; the authors state it significantly improves efficiency and flexibility. The available text does not include code, benchmark results, or model sizes, so at this stage it reads as a proposed direction rather than a ready-to-apply recipe.
How solid is it
The claims rest on the authors' own preliminary experiments plus a theoretical analysis of multi-task gradient interference. The text available does not report the datasets, tasks, model sizes, or any numeric comparison behind the stated efficiency and flexibility gains, so the size of the effect is not established by what is shown here.
Risks and caveats
The improvement in efficiency and flexibility from Parallel-RL is asserted but not quantified in the material available, and no experimental setup, benchmark, or independent replication is given. Without those details, how general or how large the effect is remains unclear from this text alone.
“Interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization.”
— from the paper's abstract