Multi-Teacher On-Policy Distillation (MOPD) enables a single student to absorb the skills of several domain teachers by learning from its own generated responses. We present Multimodal Model Multi-Teacher On-Policy Distillation Reasoner (M3OPD-Reasoner), a fully open recipe for post-training vision-language models (VLMs) over three reasoning domains: image STEM, image perception, and video. We curate two datasets, M3OPD-SFT-1.35M for supervised fine-tuning (SFT) and M3OPD-RL-193K for reinforcement learning (RL) with verifiable rewards. On these data, we compare post-training recipes that start from Qwen3.5-9B: training one model on the mixed data of all domains (MixSFT and MixRL), and training one teacher per domain with Group Sequence Policy Optimization (GSPO), directly or after an SFT warm-up, and distilling the teachers into one student with MOPD. We evaluate on 21 image STEM, image perception, and video reasoning benchmarks after we decontaminate the training data against them. The comparison establishes MOPD of teachers trained with GSPO directly from Qwen3.5-9B as the best recipe: the student closely matches each teacher on its own domain and is on average above the mixed-training baselines on the same prompts. Since MOPD already merges the domain capabilities that MixRL would otherwise have to learn, its student is a stronger starting point for RL, and we ask whether the same data can carry the model further from there: we add MixRL on top of the MOPD student, on the same prompts and with the same settings, which, to our knowledge, no prior VLM recipe has studied. The same prompts still improve the MOPD student, with no collapse, and give our main model, M3OPD-Reasoner-9B. MOPD and MixRL can therefore work as consecutive stages that get more out of the same data. We release all data, the training code, and the checkpoints of every stage.
Alex: Learning from specialists need not be the last step in training a generalist model. This study combines specialist skills, then improves the combined model using the same questions again.
Sam: Reusing questions sounds economical, but it could also mean the model has exhausted them. What did the researchers actually establish?
Alex: Their combined model still improved when correct answers earned rewards, rather than feedback from specialists. That comes from a methods preprint by Nguyen Quang Trung and colleagues at ASTAR and VinUniversity, called M three OPD-Reasoner.
Sam: So this is about how to train models, rather than a new model architecture. Who should care about the distinction?
Alex: Researchers building models that answer questions about images and video. The work suggests that combining skills and strengthening them need not happen together. We'll unpack that separation, the comparisons supporting it, and why the strongest recipe here isn't automatically the cheapest or most general.
Sam: Start with the usual recipe. What jobs is the training process trying to do at once?
Alex: Many recipes begin with supervised fine-tuning: training on example responses. They then use reinforcement learning, where responses receive rewards based on their answers. When questions from different domains are mixed together, that reward-driven stage must both combine capabilities and improve reasoning.
Sam: And the concern is that those jobs can interfere. Does the paper give a reason to separate them?
Alex: It points to earlier work reporting trade-offs between domains and conflicting training updates. One alternative is to train specialists separately, then combine their skills. Here, those specialists cover image-based maths and science, visual perception such as charts and counting, and video.
Sam: Separate specialists aren't new, though. What changes compared with previous attempts to combine them?
Alex: Earlier work already transferred several teachers into one student, including for images and video. The authors' specific addition is a separate reward-driven stage after that transfer. They say this had not previously been studied for vision-language models, which handle visual inputs and language.
Sam: Then let's make the transfer concrete. Does the student just read a collection of answers written by the experts?
Alex: No. The student writes its own response first. The question's domain determines which frozen specialist scores that response, giving feedback at every generated token. That's multi-teacher on-policy distillation: several specialists teaching on text the student actually produces.
Sam: That seems like a useful distinction. The teacher encounters the student's mistakes and choices, rather than only demonstrating its own preferred answer.
Alex: That's the rationale. Training on the student's own responses reduces the mismatch with how it behaves when answering independently. The student learns to match its routed teacher's token probabilities. During this transfer stage, there is no answer-based task reward.
Sam: How are the specialists created? If they start with different backgrounds, that could complicate the comparison.
Alex: Every model starts from the same Qwen base model. Each specialist then learns from its own domain's questions using Group Sequence Policy Optimization. That's a reinforcement-learning method comparing rewards across groups of complete responses. The authors test specialists both with and without an initial supervised warm-up.
Sam: And after the student learns from those specialists, what changes in the final stage?
Alex: The teachers stop providing feedback. The student practises across all domains, using the same questions again, with the specialists' reward rules and reinforcement-learning settings. Correctness dominates the reward, although the response must also follow the required answer format.
Sam: So teacher feedback says, roughly, “answer more like this expert.” The final stage instead asks whether the answer passes a check.
Alex: That's the distinction the recipe exploits. Matching teachers gives the student a strong starting point, but that objective doesn't independently reward going beyond them. The later checks can favour improvements that teacher matching alone would not teach.
Sam: Let's put some weight on the evidence. How broad is the test, and what stays fixed?
Alex: The transfer and mixed-domain reinforcement-learning stages use about 193,000 training prompts with checkable answers. The authors evaluate across 21 benchmarks covering the visual domains we described. They also remove training items that overlap those benchmarks through question text, image similarity, or video identifiers and frames.
Sam: When they say accuracy, are they picking the best answer from several attempts? That could change how useful the result sounds.
Alex: They generate eight responses per evaluation question and average correctness across them. It's not a score for succeeding at least once. They then average benchmark scores, so the headline result weights benchmarks equally, not every individual question equally.
Sam: What is the cleanest comparison with simply training on the mixed questions?
Alex: Starting directly from the base model, mixed-domain reinforcement learning reaches an overall accuracy of 74.4 percent. Distilling specialists trained directly from that same base reaches 75.2 percent. That's a shared-prompt comparison, not a comparison against a weaker route that first required supervised training.
Sam: The transfer advantage is modest. Does the extra reward-driven stage add anything beyond that?
Alex: It takes the overall mean to 75.9 percent. All domain averages improve, with the biggest gain on video. Most individual benchmarks improve too, although a few decline slightly. The authors also see gains when the specialists had supervised warm-ups, so the final-stage result appears in both versions.
Sam: That supports the sequence. But what makes them think the second pass is doing something different, rather than just extending training?
Alex: The training traces support their interpretation. The distilled students begin reward-driven training with accuracy that the mixed-training runs only approach later. Their response lengths also change less. The authors interpret this as evidence that transfer has already supplied skills the mixed-training runs must first acquire.
Sam: Do they show what changes in an actual answer? Training curves don't tell us whether the model uses the visual evidence better.
Alex: In a video example, the distilled student builds an incorrect account around a man firing a gun. After the final stage, the model correctly describes a woman splitting a wall with a sickle. It's an illustration, not proof of the mechanism, but it makes the change tangible.
Sam: And across the outputs more broadly, is improvement just associated with writing longer explanations?
Alex: Not on video. The final model's thinking becomes shorter, hedging becomes rare, and harder questions receive relatively more thinking than easy ones. Timestamp citations become denser, even as responses shorten. Those are output-based measures, though; they don't directly reveal the model's internal reasoning.
Sam: Here's the practical objection. Shared questions don't mean shared cost when one route trains specialists, transfers them, and then trains again.
Alex: That's a key limit. These comparisons show a stronger recipe on the same data, not equal performance per unit of computation. Every run also builds on the same base model. I'd treat the evidence as convincing within this setup, but not as proof that the ordering wins across model families or compute budgets.
Sam: Given those boundaries, who should read the full document, and where should they start?
Alex: Researchers designing multimodal post-training should start with Main results, especially the direct mixed-training comparison and the added reinforcement-learning stage. Then read Training dynamics for the proposed explanation. For implementation, the configuration appendices matter: the authors release the data, training code, and checkpoints from every stage.
Sam: And for someone who doesn't train these models, what's the result worth carrying into a conversation?
Alex: Combining skills and improving them can be separate jobs. In this study, the same questions still taught more once the model had learned from specialists.
Sam: Thanks for listening.