The recent success of agent swarms has shifted the paradigm of large language model (LLM)-based agents from single-agent workflows to multi-agent systems, highlighting the importance of agent orchestration for task decomposition and collaboration. However, existing orchestration frameworks are limited to a narrow set of modalities and struggle to generalize to more complex settings where heterogeneous modalities coexist and interact. This limitation becomes particularly pronounced in omnimodal scenarios, where tasks require the unified understanding and coordination of diverse inputs such as text, image, audio, and video. In this work, we propose Orchestra-o1, an omnimodal agent orchestration framework designed to support efficient agent collaboration across multiple modalities. Orchestra-o1 introduces a unified orchestration mechanism that enables modality-aware task decomposition, online sub-agent specialization, and parallel sub-task execution. This scalable design allows agent systems to effectively tackle complex real-world tasks involving heterogeneous information sources, surpassing the second-best approach by 10.3% accuracy on the OmniGAIA benchmark. Furthermore, we introduce decision-aligned group relative policy optimization (DA-GRPO), an efficient agentic reinforcement learning approach for training Orchestra-o1-8B, which also achieves state-of-the-art performance against all existing open-source omnimodal agents.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new framework called Orchestra-o1, which takes a fresh approach to how artificial intelligence handles complex, multi-part tasks.
Sam: Thanks, Alex. The core puzzle here is why we expect one single AI model to be a master of everything—vision, audio, code, and reasoning—all at once. This research suggests that the future of AI isn't just building a bigger or more powerful model, but building a better manager.
Alex: So is this paper arguing that current systems are hitting a wall because we're asking them to do too much at the same time?
Sam: Exactly. Right now, many systems try to be a "do-it-all" engine. Think of a film director who also tries to hold the camera, write the script, and act in every scene simultaneously. It's inefficient and prone to errors. Orchestra-o1 treats the AI as a project manager rather than a worker. It breaks a big task into smaller pieces and hands each piece to a specialist.
Alex: Okay, so it's about delegation. But how does this "manager" know which specialist should handle which part of the job?
Sam: That's the key innovation. The system uses what the researchers call "modality-aware task decomposition." Let's unpack that. Different types of information—images, audio, text, data—are called "modalities." So "modality-aware" just means the system can tell those types apart and knows which tool is built for which type.
Alex: So if I give it a video and a photo at the same time, it knows those are two different kinds of input that need two different kinds of processing?
Sam: Exactly. Say you ask: "What time does the event in this video happen, based on the landmark in this photo?" The manager recognises that identifying the landmark is a vision task, while looking up information about that landmark is a knowledge task. It then builds what the paper calls a "dependency graph"—essentially a structured plan that shows which jobs need to be done first before others can begin.
Alex: So it's not just a to-do list. It understands that you can't look up the event time until you've first identified the landmark.
Sam: Precisely. And because it maps out which tasks depend on each other, it can also spot which tasks are completely independent and run those at the same time, in parallel. It's like a construction crew where the electricians and the plumbers work simultaneously rather than waiting for each other to finish.
Alex: That makes a lot of sense for speed. But how do you actually train an AI to be a good manager? That seems like a very different skill from just knowing facts.
Sam: You're right, and that's where the second part of the paper gets interesting. The researchers developed a training method they call DA-GRPO—Decision-aligned Group Relative Policy Optimization. The name is a mouthful, but the idea is straightforward.
Alex: Walk me through it.
Sam: Think of it like a coach marking a student's work. Most AI training only checks whether the final answer is correct—pass or fail. DA-GRPO instead gives feedback on every decision the manager made along the way. Did it pick the right specialist for that step? Did it sequence the tasks correctly? Each of those choices gets evaluated, not just the end result.
Alex: So it's rewarding good process, not just good outcomes.
Sam: That's it. And there's a practical reason for doing it this way. Evaluating decisions on paper—offline, without actually running every sub-task—is far cheaper than running the full system every time during training. It also gives what the paper calls "dense feedback," meaning the model gets precise information about exactly where its plan went wrong, rather than just knowing the final answer was off.
Alex: It's like the difference between a teacher writing notes in the margin of every page versus just writing "wrong" at the end of the essay.
Sam: That's a good way to put it. The other piece of this training is something called trajectory alignment. The researchers take high-quality examples of how a top-tier model solves a complex task—the whole sequence of decisions it makes—and use those as a template. The smaller model then learns to match that pattern of reasoning, not just the final answer.
Alex: So the smaller model is essentially learning to think like a more capable one?
Sam: Yes. By studying how an expert breaks a problem apart and assigns the right tools, the model internalises that decision-making strategy. The result is that a relatively compact model can act as a capable orchestrator, even when the underlying tasks are complex.
Alex: And does that actually hold up when you test it?
Sam: The results are notable. On a benchmark called OmniGAIA—which tests AI systems on complex, multi-modal tasks—the trained orchestrator outperformed larger baseline models. And in tests using proprietary systems, Orchestra-o1 beat the strongest single-model approach by more than ten percentage points in accuracy. That's a meaningful gap.
Alex: So the lesson isn't just "make the model bigger." It's "make the manager smarter."
Sam: That's the core takeaway. By separating the high-level planning from the low-level execution, the system avoids the bottleneck of one model trying to handle everything at once. Each specialist does what it's built for, and the orchestrator's job is simply to coordinate them well.
Alex: Are there limits to how far this approach can go?
Sam: There's an important one. The orchestrator can only work with what its sub-agents give it. If the specialist tools are weak or produce unreliable outputs, no amount of clever planning at the top level can fix that. The quality of the whole system depends on the quality of its parts.
Alex: So it's a strong architecture, but it doesn't solve every problem on its own.
Sam: Right. What it does solve is the coordination problem—the inefficiency of asking one model to be everything to everyone. For tasks that genuinely require multiple types of reasoning working together, this kind of manager-and-specialist structure appears to be a more practical path forward than simply scaling up a single model.
Alex: That's a useful frame for thinking about where AI development is heading. Rather than one ever-larger brain, perhaps a well-organised team. Thanks for walking us through it, Sam.
Sam: Thanks, Alex. It's a good reminder that in complex systems, how you organise the work often matters as much as the raw capability of the workers.
Alex: Thanks for listening to ResearchPod.