Co-RL trains diverse model cohorts to reason without labeled data

Reinforcement learning has become one of the most effective ways to improve reasoning in large language models and vision-language models, but its biggest gains still depend on ground-truth supervision, such as a verifiable reward that checks an answer against a known correct one. That kind of labeled data is costly to produce, and it grows scarcer as models' reasoning abilities move past what humans can reliably judge. Self-rewarding reinforcement learning tries to get around this by having a model generate its own reward signal from its own answers, but training on nothing but a model's own feedback can reinforce its existing biases and weak habits, narrow the range of its responses, and eventually collapse into repetitive, homogenized behavior.

The researchers introduce Co-RL, a framework built on cooperative multi-agent training rather than self-rewarding or ground-truth supervision. Several separate models, none of which share parameters with each other, are optimized at the same time through reinforcement learning, and each model's reward comes from its peers rather than from labeled answers or its own judgment of itself. The central finding is that making that cohort more diverse, by mixing different model families, different model sizes, and differently rephrased versions of the same training samples, cuts down the correlated errors that would otherwise let the whole group reinforce the same mistakes and drift toward the kind of collapse seen in self-rewarding setups. That added diversity consistently improves reasoning performance, keeps the models' responses varied instead of converging on one pattern, and holds off training collapse.

Tested across both text-only and multimodal settings, Co-RL consistently outperforms the base models it starts from and prior label-free approaches, and it matches or surpasses methods that rely on supervised, ground-truth training, despite never seeing any ground-truth labels itself. The gains are concrete: an average improvement of 3.0-8.6% across seven text-only benchmarks for language models, and 2.3-7.2% across four multimodal benchmarks for vision-language models. Code implementing Co-RL has been released on GitHub.

Key facts

Co-RL trains several separate models, sharing no parameters, at the same time through reinforcement learning, with each model's reward supplied by its peers rather than by labeled data.
Making the training cohort more diverse, through different model families, different model sizes, and rephrased training samples, cuts the correlated errors that normally push self-rewarding methods toward collapse.
Co-RL lifts reasoning performance for language models by an average of 3.0-8.6% across seven text-only benchmarks.
On four multimodal benchmarks, Co-RL improves vision-language models by an average of 2.3-7.2%.
Co-RL matches or surpasses supervised training methods that use ground-truth labels, despite using none itself, and its code has been released on GitHub.

Why it matters

Reinforcement learning drives much of the recent progress in reasoning models, but the strongest results still lean on ground-truth supervision, such as a verifiable reward that checks an answer against a known correct one. That kind of labeled data is expensive to produce, and it becomes scarcer exactly as models' reasoning starts to exceed what a human reviewer can reliably judge. Letting a model reward itself removes the need for labels, but training on nothing except a model's own feedback can reinforce whatever biases it already has, narrow its range of responses, and eventually lead to homogenized, repetitive output and training collapse. Co-RL's contribution is showing that training a cohort of different, decoupled models against each other, rather than one model against itself, keeps that collapse from setting in while still improving reasoning without any ground-truth labels at all.

Who it affects

This is aimed at researchers and labs building reasoning-focused language and vision-language models who currently depend on curated, verifiable-reward datasets to push reasoning further, as well as anyone running self-rewarding reinforcement learning pipelines that have hit the homogenization or collapse problems the paper describes. The abstract names no individual authors, institutions, or affiliations behind the work.

How to use it

This is a training method described in a paper, not a hosted product. The authors say code implementing Co-RL is available on GitHub, so a team that wants to reproduce the approach or build on it has an actual repository to start from rather than having to reimplement the method from the abstract alone.

How solid is it

The claims rest on the authors' own reported results: average gains of 3.0-8.6% across seven text-only benchmarks for language models and 2.3-7.2% across four multimodal benchmarks for vision-language models, measured against base models and prior label-free methods. The abstract states these as relative gains only, without giving an absolute accuracy or performance level for Co-RL or the methods it's compared against. On Hugging Face's papers board the submission has drawn modest attention so far: 86 points and two comments, which reflects visibility rather than outside verification of the results.

Risks and caveats

The paper's own language treats the collapse risk as a possibility rather than a guarantee: it says self-rewarding training 'can' reinforce bias and homogenize outputs, not that it always does, and Co-RL is built specifically to counter that failure mode by replacing a single self-rewarding model with a diverse cohort. The performance numbers are the authors' own reported results, measured as relative gains against base models and prior methods rather than as absolute accuracy levels, so the headline percentages describe improvement over a baseline rather than an absolute quality bar.