Glonce

Co-RL trains AI models to reason without labeled data

researchThe Glonce editorsHugging Face🔥59Glonce rating59Impact68Novelty77Relevance100Surprise60Players45Credibility87
Co-RL trains AI models to reason without labeled data

Researchers have introduced Co-RL, a multi-agent reinforcement learning framework built to improve reasoning in large language models (LLMs) and vision-language models (VLMs) without any ground-truth labels. Standard reinforcement learning for reasoning usually depends on verifiable rewards, meaning a human or an automated checker confirms whether an answer is correct. Those annotations are expensive to produce and grow scarcer as models tackle problems people can no longer reliably judge themselves. An alternative, self-rewarding RL, lets a model generate its own reward signal from its own outputs, cutting the need for outside supervision. The authors argue this shortcut carries a real risk: training a model solely on feedback it generates for itself can reinforce its existing biases and weak habits, narrow the range of its responses, and in the worst case collapse training into homogenized output.

Co-RL avoids that failure mode by training several separate models at once, models that share no parameters with each other, with each one drawing its reward signal from its peers rather than from itself. The authors show that increasing the diversity of this cohort, by mixing different model families, different model sizes, and rephrased versions of the same training samples, reduces the correlated errors that otherwise drive a self-reinforcing feedback loop. That diversity is what keeps the peer group from converging on the same blind spots as training proceeds, preserving response variety and avoiding collapse.

Tested across text-only and multimodal domains, Co-RL consistently outperformed both the base models it started from and prior label-free training methods, and it matched or surpassed supervised methods that do use ground-truth labels. On seven text-only benchmarks for LLMs, Co-RL delivered average gains of 3.0 to 8.6 percent over baselines; on four multimodal benchmarks for VLMs, average gains ran 2.3 to 7.2 percent. The source gives only these aggregate ranges, not a per-benchmark breakdown, and does not state dataset sizes, model sizes, or compute budgets. Code for the framework is released on GitHub under the project name Co-RL.

Key facts

  • Co-RL trains multiple decoupled models simultaneously, with each model's reward signal coming from its peers rather than from itself or from human labels.
  • Increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing training collapse.
  • Co-RL improved reasoning by an average of 3.0 to 8.6 percent across seven text-only benchmarks for LLMs.
  • Co-RL improved reasoning by an average of 2.3 to 7.2 percent across four multimodal benchmarks for VLMs.
  • Co-RL matched or surpassed supervised methods despite using no ground-truth labels at all, and its code is public on GitHub.

Why it matters

Reinforcement learning has become the main lever for improving reasoning in LLMs and VLMs, but the strongest results still lean on verifiable, ground-truth rewards, annotations that are costly to produce and increasingly scarce once a model's reasoning outruns what a human can reliably check. Self-rewarding RL, where a model scores its own outputs, promised a way around that bottleneck, but the authors show it tends to reinforce a model's own biases and can collapse training into repetitive, homogenized responses. Co-RL's cooperative multi-agent setup, where independent models reward each other instead of themselves, offers a route to keep improving reasoning as verified labels become harder to get.

Who it affects

Teams building LLMs or VLMs that want to keep pushing reasoning performance as ground-truth supervision gets more expensive or simply runs out; anyone already using self-rewarding RL and running into the bias-reinforcement or training-collapse problems the authors describe.

How to use it

The authors release code for Co-RL on GitHub under the project name Co-RL. The source text gives no pricing, license terms, or setup instructions beyond that the framework requires training a cohort of several decoupled models at once, and that mixing model families, sizes, and rephrased training samples is what makes the diversity effect work; no further deployment detail is given.

How solid is it

The reported gains, 3.0 to 8.6 percent on seven text-only benchmarks and 2.3 to 7.2 percent on four multimodal benchmarks, are stated as averages across each benchmark set and held consistently enough for the authors to claim Co-RL matches or beats supervised baselines with no labels at all. The source, however, gives only these aggregate ranges: it does not break results out per individual benchmark, and it states no dataset sizes, model sizes, compute budgets, or publication venue, so the scale of the underlying experiments cannot be assessed from the text alone.

Risks and caveats

The mechanism by which peer models compute reward signals for each other is not detailed beyond the general statement that rewards are 'derived from their peers', leaving a key algorithmic step unclear. Without the missing per-benchmark numbers, it is not possible to tell whether the reported gains are spread evenly or concentrated in a few tasks. The framework's core premise, that diversity alone prevents collapse, is demonstrated on the tested cohort configurations only; how it holds up at larger scale or with less diverse model pools is not addressed in the source.