Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL | Yunhao Yang et al. | ResearchPod

Transcript: Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

Alex: Welcome to another episode of ResearchPod.

Sam: So this paper is asking whether a computer model can get smarter at math and coding entirely on its own — without a human grading its work?

Alex: That's the central question. The problem is that current training methods depend on expensive, human-verified answers, and those are becoming harder to come by. But if you let a model train purely on its own outputs, it tends to reinforce its own mistakes rather than correct them.

Sam: So it gets stuck in its own blind spots.

Alex: Exactly. The study introduces a framework called CO-RL — cooperative reinforcement learning — that tries to solve this by having models supervise each other instead of themselves.

Sam: How does that actually work?

Alex: Think of two students studying for the same exam. Instead of checking your own homework, you swap papers. You grade your partner's work, and they grade yours. Because you've studied independently, you tend to catch different mistakes. That's the core idea here.

Sam: And that outside perspective is what breaks the feedback loop?

Alex: Right. When a model grades its own answers, it tends to reward the same kinds of responses it already favors — even when those responses are wrong. A second model, trained differently, has a different set of blind spots. So when it grades the first model's work, it provides a correction signal that the first model couldn't generate for itself. The paper calls this decorrelated supervision.

Sam: Does that mean any two models can pair up and benefit?

Alex: Not quite. The paper shows that the diversity between the two models matters a great deal. When models come from different families — meaning they were built and trained using different approaches — their errors tend not to overlap. One can catch what the other would normally miss. Pairing identical models doesn't produce the same benefit, because they tend to make the same mistakes.

Sam: So the disagreement between them is actually useful.

Alex: Precisely. In practice, each model generates a set of answers, and the pair uses a majority vote across those answers to form what the paper calls a pseudo-label — essentially a best guess at the correct answer, agreed upon collectively. A response gets a positive signal if it matches the peer's majority vote, and nothing otherwise.

Sam: How does that compare to older approaches that use a separate model as a judge?

Alex: The paper tests CO-RL against those judge-based systems — where a third model evaluates the outputs — and finds the cooperative peer approach performs better on average across seven standard benchmarks, while actually using fewer models and no separate judge. It's a more efficient setup that still produces stronger results.

Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

Yunhao Yang, Yuexin Bian, Yunjie Tian, Di Fu, Tianjin Huang, Yuanyuan Shi, Ziang Xiao, Nuno Vasconcelos, Yijiang Li

Aug 18, 2026·4:35·arxiv·Original Paper

Co-RLUnsupervised ReasoningCohort DiversitySelf-Rewarding RL LimitationsCO-RLDecorrelated Supervision

Listen

5 min

Briefing

Research Question and Motivation

Reinforcement learning with verifiable rewards has proven highly effective for improving reasoning capabilities in language and vision-language models, but it relies heavily on costly and scarce ground-truth annotations. While self-rewarding reinforcement learning attempts to bypass this limitation by using a model's own completions to generate reward signals, training solely on self-generated feedback often reinforces existing biases, reduces response diversity, and leads to training collapse. This paper investigates whether unsupervised reasoning can successfully emerge without any ground-truth supervision by leveraging cooperative multi-agent training.

The Co-Reinforcement Learning Framework

The authors introduce Co-Reinforcement Learning (Co-RL), a label-free multi-agent reinforcement learning method. In this framework, multiple decoupled agents that share no parameters are simultaneously optimized through reinforcement learning using rewards derived from their peers. Given an unlabeled prompt, each agent independently samples multiple completions and aggregates them into a pseudo-answer using majority voting. The completions of one agent are then rewarded against a peer agent's pseudo-answer, and these peer-derived rewards drive policy optimization.

Cohort Diversity and Error Decorrelation

A central insight of the paper is that what a cohort teaches depends on how different its mistakes are. Highly similar models tend to make correlated errors and reinforce incorrect answers. To maximize error decorrelation, Co-RL leverages heterogeneous model families, varying model sizes, and independent policy optimization. By drawing supervision from an independently updated partner rather than the model's own view, Co-RL breaks the self-reinforcing feedback loop. Experiments across text-only benchmarks for large language models and multimodal benchmarks for vision-language models demonstrate that Co-RL consistently outperforms base models and prior label-free approaches while matching or surpassing supervised methods.

Key Terms and Definitions

Co-Reinforcement Learning (Co-RL) — A label-free multi-agent reinforcement learning framework where multiple decoupled models are optimized simultaneously using peer-derived reward signals.
Group Relative Policy Optimization (GRPO) — A policy optimization algorithm that estimates response advantages by normalizing rewards within a group without requiring a separately trained critic model.
Self-Rewarding RL — A training paradigm where a model derives reward signals solely from its own completions and internal predictions.
Cohort Diversity — The inclusion of heterogeneous model families, distinct model sizes, and varied input formulations to reduce correlated errors among agents.
Pseudo-Label — An answer derived from a peer agent's majority vote that serves as a supervision target in the absence of ground-truth labels.

Abstract

Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs. Code is available at https://github.com/DrStranded/Co-RL.

Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL: Research Paper Podcast