Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs. Code is available at https://github.com/DrStranded/Co-RL.
Alex: Welcome to another episode of ResearchPod.
Sam: So this paper is asking whether a computer model can get smarter at math and coding entirely on its own — without a human grading its work?
Alex: That's the central question. The problem is that current training methods depend on expensive, human-verified answers, and those are becoming harder to come by. But if you let a model train purely on its own outputs, it tends to reinforce its own mistakes rather than correct them.
Sam: So it gets stuck in its own blind spots.
Alex: Exactly. The study introduces a framework called CO-RL — cooperative reinforcement learning — that tries to solve this by having models supervise each other instead of themselves.
Sam: How does that actually work?
Alex: Think of two students studying for the same exam. Instead of checking your own homework, you swap papers. You grade your partner's work, and they grade yours. Because you've studied independently, you tend to catch different mistakes. That's the core idea here.
Sam: And that outside perspective is what breaks the feedback loop?
Alex: Right. When a model grades its own answers, it tends to reward the same kinds of responses it already favors — even when those responses are wrong. A second model, trained differently, has a different set of blind spots. So when it grades the first model's work, it provides a correction signal that the first model couldn't generate for itself. The paper calls this decorrelated supervision.
Sam: Does that mean any two models can pair up and benefit?
Alex: Not quite. The paper shows that the diversity between the two models matters a great deal. When models come from different families — meaning they were built and trained using different approaches — their errors tend not to overlap. One can catch what the other would normally miss. Pairing identical models doesn't produce the same benefit, because they tend to make the same mistakes.
Sam: So the disagreement between them is actually useful.
Alex: Precisely. In practice, each model generates a set of answers, and the pair uses a majority vote across those answers to form what the paper calls a pseudo-label — essentially a best guess at the correct answer, agreed upon collectively. A response gets a positive signal if it matches the peer's majority vote, and nothing otherwise.
Sam: How does that compare to older approaches that use a separate model as a judge?
Alex: The paper tests CO-RL against those judge-based systems — where a third model evaluates the outputs — and finds the cooperative peer approach performs better on average across seven standard benchmarks, while actually using fewer models and no separate judge. It's a more efficient setup that still produces stronger results.
Sam: Did they test what happens when you add a third model into the mix?
Alex: They did. Jointly training three distinct models in a single run — with no human-verified answers provided at any point — matched or outperformed models that had been trained directly on human-labeled data. The collective voting among three different models acts as a reliable filter, catching errors that any single model would miss.
Sam: That's a meaningful result. What about tasks that involve images, not just text?
Alex: The authors extended the framework to what are called vision-language models — systems that process both text and images together. The cooperative supervision held up there as well. Across multimodal tasks, the peer-based approach consistently outperformed standard self-rewarding baselines and occasionally surpassed training with direct human supervision.
Sam: And what about training stability? I know single-model self-rewarding systems can degrade over time.
Alex: That's one of the more notable findings. Single-model systems often collapse — the reward signal becomes unstable, and the model's outputs drift in unhelpful directions. The peer-supervised approach maintained stable reward variation and consistent output lengths throughout training, avoiding that kind of collapse.
Sam: So what's the practical cost of running all this?
Alex: The main trade-off is computational. You have to maintain and jointly train multiple model instances at the same time, which requires more memory and processing power than training a single model. The authors are transparent about that in the paper.
Sam: But the performance gains suggest the extra overhead is worth it?
Alex: The benchmark results point in that direction, yes. And when you zoom out, the broader implication is significant. If models can reliably teach and verify one another without human intervention, it opens a path toward learning systems that don't depend on a continuous supply of human-labeled data — which is currently one of the main bottlenecks in AI development.
Sam: So peer supervision might be a meaningful step toward more self-sufficient AI systems.
Alex: That's what the evidence suggests. It's early work, and the authors are careful not to overclaim, but the direction is clear: diverse models checking each other's reasoning can produce results that neither could achieve alone.
Sam: That's a genuinely interesting finding. Thanks for listening to ResearchPod.