We introduce Motif 3, a decoder-only Mixture-of-Experts language model with 314 billion total parameters and 13.2 billion activated per token. Each sparse MoE layer contains 384 routed experts, with eight selected per token. This fine-grained sparsity provides substantial expert capacity while limiting computation. Motif 3 is built around Grouped Differential Latent Attention (GDLA), which integrates grouped differential attention with the compressed key-value representation of Multi-head Latent Attention. The architecture further incorporates modified manifold-constrained hyper-connections, Expert Specific PolyNorm activations, and multi-token prediction to improve optimization stability, expert specialization, and inference efficiency. We pretrain Motif 3 on approximately 12.5 trillion tokens spanning web documents, STEM, code, mathematics, multilingual content, and domain-specialized corpora. Expert-balancing and numerical-stabilization techniques support stable training at scale, while selective MXFP8 computation and communication, memory-efficient fused kernels, and window-aware context parallelism enable training with context lengths up to 256K tokens. Our post-training pipeline combines general supervised fine-tuning, six specialist teachers trained with reinforcement learning, a software-engineering teacher trained with supervised fine-tuning, and Multi-teacher On-Policy Distillation. The resulting unified model consolidates complementary capabilities in reasoning, coding, tool use, professional work, long-context understanding, calibrated abstention, and instruction following. Across a broad evaluation suite, Motif 3 demonstrates competitive performance against leading open weight models, including strong results on long-horizon agentic tasks, mathematical reasoning, scientific knowledge, and hallucination-sensitive evaluation.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at a paper about Motif 3, a very large language model designed to stay broadly capable without spending full effort on every word it processes. The central idea is to give the model a large pool of specialist skills, but only draw on a small slice of them at any one time.
Alex: So it's big, but it tries not to act big on every single step?
Sam: That's the right way to put it. Think of it like a hospital with hundreds of specialist doctors. When a patient arrives, you don't call in every specialist at once — you route them to the two or three most relevant ones. Motif 3 works on a similar principle. Each time it processes a small piece of text, called a token, it selects only eight specialists from a pool of 384. The rest stay idle. That keeps the actual work manageable, even though the total capacity is very large.
Alex: Why does having that many specialists matter if most of them are sitting out?
Sam: Because different tokens need different kinds of knowledge. A token in a math problem needs different handling than a token in a piece of poetry or a line of code. Having a large pool means there's likely a good match for whatever comes in. The routing step is what makes that practical — it keeps the cost low by only activating the right few, rather than all of them.
Alex: But doesn't that create a risk? What if the router just keeps picking the same favorites?
Sam: That's one of the paper's main concerns. If the same few experts keep getting selected, they get more practice, become even more attractive to the router, and the others gradually stop being useful. It's a bit like a classroom where the teacher only ever calls on the same three students — eventually the rest stop paying attention. To prevent that, the authors use a load-balancing penalty early in training, so the router is pushed to spread its choices more evenly. They also inject temporary random noise so the system explores a wider range of experts before it settles into habits.
Alex: And they keep the routing decisions themselves precise, even when other parts of the computation are simplified?
Sam: Exactly. Routing decisions and some of the most sensitive calculations are kept in full precision — meaning the model doesn't cut corners there. For the heavy expert computation, they use a lower-cost numerical format to save memory and speed things up. Think of it like using a lighter backpack for the parts of a trip that don't carry anything fragile. You save effort where it's safe, but protect the things that matter most.
Alex: That makes sense. So the architecture is designed to be selective. What about the training data that feeds into it?
Sam: The model is pretrained on roughly 12.5 trillion tokens drawn from web text, science, code, mathematics, multilingual material, and some specialized domains. The logic is straightforward: a model with this much capacity needs a broad diet, or it risks becoming good at only a narrow slice of language while the rest of its capacity goes to waste.
Alex: And keeping that much training stable must be its own challenge.
Sam: It is. The authors describe a combination of balancing rules, numerical safeguards, and careful choices about which parts of the model to keep fixed during later training stages. The goal is to stop different parts of the system from pulling against each other as training progresses.
Alex: So after all that pretraining, how do they shape the model's final behavior?
Sam: In stages. First comes supervised fine-tuning, where the model is shown examples of the kinds of answers it should produce — this teaches it the style and format expected of it. Then the authors train a set of specialist teacher models, each focused on a particular domain: tool use, software engineering, long-context reasoning, mathematics, and others. Finally, they use a process called distillation, where the student model learns from all those teachers at once, absorbing their strengths and folding them into a single system.
Alex: So it's not copying one source. It's more like combining lessons from several tutors into one voice.
Sam: That's a fair description. The paper's claim is that this staged approach helps the model keep useful behavior across many different kinds of tasks, without needing separate models for each one at deployment.
Alex: And does the evidence support that?
Sam: To a meaningful degree, yes — though the paper is careful about the limits. The strongest results are on tasks that require multi-step action, like working in a terminal or using external tools. It's also competitive on coding, reasoning, long-context understanding, and instruction following. But the authors are clear that performance is weaker on some specialized scientific reasoning and scientific coding tasks. And because Motif 3 is primarily a text system, it doesn't handle visual input.
Alex: So there are real gaps.
Sam: There are. The paper also notes that real-world use can be messier than controlled evaluations, particularly for long-horizon tasks that require planning and recovery over extended sequences. The authors present the results as meaningful progress, not a final answer.
Alex: That's a measured way to put it. The design is coherent, the results are encouraging in places, but the work isn't finished.
Sam: That's the fair reading. Motif 3 is a careful piece of engineering that tries to show sparse expert routing, broad pretraining, and staged post-training can work together in one system. The contribution is a coherent approach to building a large model that's more efficient and broadly useful — not a claim that every problem is solved.
Alex: Thanks for walking through it. And thanks to everyone listening to ResearchPod.