Fan Zhang, Vireo Zhang, Shengju Qian, Haoxuan Li, Hao Wu, Jinyang Wu, Donghao Zhou, Zhihong Zhu, Zheng Lian, Xin Wang, Pheng-Ann Heng
6 min
Abstract
The recent success of agent swarms has shifted the paradigm of large language model (LLM)-based agents from single-agent workflows to multi-agent systems, highlighting the importance of agent orchestration for task decomposition and collaboration. However, existing orchestration frameworks are limited to a narrow set of modalities and struggle to generalize to more complex settings where heterogeneous modalities coexist and interact. This limitation becomes particularly pronounced in omnimodal scenarios, where tasks require the unified understanding and coordination of diverse inputs such as text, image, audio, and video. In this work, we propose Orchestra-o1, an omnimodal agent orchestration framework designed to support efficient agent collaboration across multiple modalities. Orchestra-o1 introduces a unified orchestration mechanism that enables modality-aware task decomposition, online sub-agent specialization, and parallel sub-task execution. This scalable design allows agent systems to effectively tackle complex real-world tasks involving heterogeneous information sources, surpassing the second-best approach by 10.3% accuracy on the OmniGAIA benchmark. Furthermore, we introduce decision-aligned group relative policy optimization (DA-GRPO), an efficient agentic reinforcement learning approach for training Orchestra-o1-8B, which also achieves state-of-the-art performance against all existing open-source omnimodal agents.
Alex: Walk me through it.
Sam: Think of it like a coach marking a student's work. Most AI training only checks whether the final answer is correct—pass or fail. DA-GRPO instead gives feedback on every decision the manager made along the way. Did it pick the right specialist for that step? Did it sequence the tasks correctly? Each of those choices gets evaluated, not just the end result.
Alex: So it's rewarding good process, not just good outcomes.
Sam: That's it. And there's a practical reason for doing it this way. Evaluating decisions on paper—offline, without actually running every sub-task—is far cheaper than running the full system every time during training. It also gives what the paper calls "dense feedback," meaning the model gets precise information about exactly where its plan went wrong, rather than just knowing the final answer was off.
Alex: It's like the difference between a teacher writing notes in the margin of every page versus just writing "wrong" at the end of the essay.
Sam: That's a good way to put it. The other piece of this training is something called trajectory alignment. The researchers take high-quality examples of how a top-tier model solves a complex task—the whole sequence of decisions it makes—and use those as a template. The smaller model then learns to match that pattern of reasoning, not just the final answer.
Alex: So the smaller model is essentially learning to think like a more capable one?
Sam: Yes. By studying how an expert breaks a problem apart and assigns the right tools, the model internalises that decision-making strategy. The result is that a relatively compact model can act as a capable orchestrator, even when the underlying tasks are complex.
Alex: And does that actually hold up when you test it?
Sam: The results are notable. On a benchmark called OmniGAIA—which tests AI systems on complex, multi-modal tasks—the trained orchestrator outperformed larger baseline models. And in tests using proprietary systems, Orchestra-o1 beat the strongest single-model approach by more than ten percentage points in accuracy. That's a meaningful gap.
Alex: So the lesson isn't just "make the model bigger." It's "make the manager smarter."
Sam: That's the core takeaway. By separating the high-level planning from the low-level execution, the system avoids the bottleneck of one model trying to handle everything at once. Each specialist does what it's built for, and the orchestrator's job is simply to coordinate them well.
Alex: Are there limits to how far this approach can go?
Sam: There's an important one. The orchestrator can only work with what its sub-agents give it. If the specialist tools are weak or produce unreliable outputs, no amount of clever planning at the top level can fix that. The quality of the whole system depends on the quality of its parts.
Alex: So it's a strong architecture, but it doesn't solve every problem on its own.
Sam: Right. What it does solve is the coordination problem—the inefficiency of asking one model to be everything to everyone. For tasks that genuinely require multiple types of reasoning working together, this kind of manager-and-specialist structure appears to be a more practical path forward than simply scaling up a single model.
Alex: That's a useful frame for thinking about where AI development is heading. Rather than one ever-larger brain, perhaps a well-organised team. Thanks for walking us through it, Sam.
Sam: Thanks, Alex. It's a good reminder that in complex systems, how you organise the work often matters as much as the raw capability of the workers.
Alex: Thanks for listening to ResearchPod.