Yunhao Yang, Yuexin Bian, Yunjie Tian, Di Fu, Tianjin Huang, Yuanyuan Shi, Ziang Xiao, Nuno Vasconcelos, Yijiang Li
5 min
Abstract
Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs. Code is available at https://github.com/DrStranded/Co-RL.
Sam: Did they test what happens when you add a third model into the mix?
Alex: They did. Jointly training three distinct models in a single run — with no human-verified answers provided at any point — matched or outperformed models that had been trained directly on human-labeled data. The collective voting among three different models acts as a reliable filter, catching errors that any single model would miss.
Sam: That's a meaningful result. What about tasks that involve images, not just text?
Alex: The authors extended the framework to what are called vision-language models — systems that process both text and images together. The cooperative supervision held up there as well. Across multimodal tasks, the peer-based approach consistently outperformed standard self-rewarding baselines and occasionally surpassed training with direct human supervision.
Sam: And what about training stability? I know single-model self-rewarding systems can degrade over time.
Alex: That's one of the more notable findings. Single-model systems often collapse — the reward signal becomes unstable, and the model's outputs drift in unhelpful directions. The peer-supervised approach maintained stable reward variation and consistent output lengths throughout training, avoiding that kind of collapse.
Sam: So what's the practical cost of running all this?
Alex: The main trade-off is computational. You have to maintain and jointly train multiple model instances at the same time, which requires more memory and processing power than training a single model. The authors are transparent about that in the paper.
Sam: But the performance gains suggest the extra overhead is worth it?
Alex: The benchmark results point in that direction, yes. And when you zoom out, the broader implication is significant. If models can reliably teach and verify one another without human intervention, it opens a path toward learning systems that don't depend on a continuous supply of human-labeled data — which is currently one of the main bottlenecks in AI development.
Sam: So peer supervision might be a meaningful step toward more self-sufficient AI systems.
Alex: That's what the evidence suggests. It's early work, and the authors are careful not to overclaim, but the direction is clear: diverse models checking each other's reasoning can produce results that neither could achieve alone.
Sam: That's a genuinely interesting finding. Thanks for listening to ResearchPod.