Motif Technologies
5 min
Abstract
We introduce Motif 3, a decoder-only Mixture-of-Experts language model with 314 billion total parameters and 13.2 billion activated per token. Each sparse MoE layer contains 384 routed experts, with eight selected per token. This fine-grained sparsity provides substantial expert capacity while limiting computation. Motif 3 is built around Grouped Differential Latent Attention (GDLA), which integrates grouped differential attention with the compressed key-value representation of Multi-head Latent Attention. The architecture further incorporates modified manifold-constrained hyper-connections, Expert Specific PolyNorm activations, and multi-token prediction to improve optimization stability, expert specialization, and inference efficiency. We pretrain Motif 3 on approximately 12.5 trillion tokens spanning web documents, STEM, code, mathematics, multilingual content, and domain-specialized corpora. Expert-balancing and numerical-stabilization techniques support stable training at scale, while selective MXFP8 computation and communication, memory-efficient fused kernels, and window-aware context parallelism enable training with context lengths up to 256K tokens. Our post-training pipeline combines general supervised fine-tuning, six specialist teachers trained with reinforcement learning, a software-engineering teacher trained with supervised fine-tuning, and Multi-teacher On-Policy Distillation. The resulting unified model consolidates complementary capabilities in reasoning, coding, tool use, professional work, long-context understanding, calibrated abstention, and instruction following. Across a broad evaluation suite, Motif 3 demonstrates competitive performance against leading open weight models, including strong results on long-horizon agentic tasks, mathematical reasoning, scientific knowledge, and hallucination-sensitive evaluation.
Sam: The model is pretrained on roughly 12.5 trillion tokens drawn from web text, science, code, mathematics, multilingual material, and some specialized domains. The logic is straightforward: a model with this much capacity needs a broad diet, or it risks becoming good at only a narrow slice of language while the rest of its capacity goes to waste.
Alex: And keeping that much training stable must be its own challenge.
Sam: It is. The authors describe a combination of balancing rules, numerical safeguards, and careful choices about which parts of the model to keep fixed during later training stages. The goal is to stop different parts of the system from pulling against each other as training progresses.
Alex: So after all that pretraining, how do they shape the model's final behavior?
Sam: In stages. First comes supervised fine-tuning, where the model is shown examples of the kinds of answers it should produce — this teaches it the style and format expected of it. Then the authors train a set of specialist teacher models, each focused on a particular domain: tool use, software engineering, long-context reasoning, mathematics, and others. Finally, they use a process called distillation, where the student model learns from all those teachers at once, absorbing their strengths and folding them into a single system.
Alex: So it's not copying one source. It's more like combining lessons from several tutors into one voice.
Sam: That's a fair description. The paper's claim is that this staged approach helps the model keep useful behavior across many different kinds of tasks, without needing separate models for each one at deployment.
Alex: And does the evidence support that?
Sam: To a meaningful degree, yes — though the paper is careful about the limits. The strongest results are on tasks that require multi-step action, like working in a terminal or using external tools. It's also competitive on coding, reasoning, long-context understanding, and instruction following. But the authors are clear that performance is weaker on some specialized scientific reasoning and scientific coding tasks. And because Motif 3 is primarily a text system, it doesn't handle visual input.
Alex: So there are real gaps.
Sam: There are. The paper also notes that real-world use can be messier than controlled evaluations, particularly for long-horizon tasks that require planning and recovery over extended sequences. The authors present the results as meaningful progress, not a final answer.
Alex: That's a measured way to put it. The design is coherent, the results are encouraging in places, but the work isn't finished.
Sam: That's the fair reading. Motif 3 is a careful piece of engineering that tries to show sparse expert routing, broad pretraining, and staged post-training can work together in one system. The contribution is a coherent approach to building a large model that's more efficient and broadly useful — not a claim that every problem is solved.
Alex: Thanks for walking through it. And thanks to everyone listening to ResearchPod.