MADA-RL: Multi-Agent Debate-Aware Reinforcement Learning for Parameter-Efficient Reasoning in Compact Models

Large language models (LLMs) have demonstrated strong reasoning capabilities, but these often come at a high computational cost, making them difficult to deploy in resource-constrained environments with models under 4 billion parameters. This paper investigates how to improve the reasoning performance of compact models through a parameter-efficient post-training framework that combines multi-agent debate with reinforcement learning (RL).

The authors introduce MADA-RL, a framework that specializes compact models into two distinct roles: generators and critics. The core innovation is a counterfactual critic advantage, which redefines the RL reward signal. Instead of simply rewarding a critic for producing a correct answer, the model is rewarded for its performance relative to the average accuracy of a generator ensemble. This forces the critic to learn how to correct errors made by the generators rather than merely imitating them. The framework uses Low-Rank Adaptation (LoRA) to fine-tune only a small subset of parameters, keeping the base model weights frozen to maintain efficiency. At deployment, these specialized agents interact in a multi-round debate protocol.

Across five mathematical reasoning benchmarks, MADA-RL improved the accuracy of the DeepSeek-R1-Distill-Qwen-1.5B model from 39.9% to 41.9%. This gain was achieved using 16 times fewer trainable parameters than standard full-model fine-tuning baselines. A controlled analysis confirmed that the counterfactual advantage was the primary driver of these gains, as it resulted in the highest critic improvement rate among all evaluated models. While MADA-RL approaches the performance of larger, more data-intensive models like DeepScaleR and STILL-3, it does not surpass them, highlighting a trade-off between training efficiency and absolute reasoning capability.