Marko Kojic, Ivan Bondyrev, Aral de Moor, Joseph Shtok, Petr Borovlev, Kseniia Lysaniuk, Madeeswaran Kannan, Ivan Dolgov, Nikita Pavlichenko
6 min
Abstract
We present Mellum 2, an open-weight 12B-parameter Mixture-of-Experts (MoE) language model with 2.5B active parameters per token. Mellum 2 is a general-purpose language model specialized in software engineering, spanning code generation and editing, debugging, multi-step reasoning, tool use and function calling, agentic coding, and conversational programming assistance, and it is the successor to the completion-focused 4B dense Mellum model. The architecture builds on the Mixture-of-Experts (64 experts, 8 active) and combines Grouped-Query Attention with 4 KV heads, Sliding Window Attention on three of every four layers, and a single Multi-Token Prediction head that doubles as both an auxiliary pre-training objective and a built-in draft model for speculative decoding; each choice was validated by ablation with inference efficiency on commodity GPUs as a design constraint. Pre-training spans approximately 10.6 trillion tokens through a three-phase curriculum that progressively shifts the mixture from diverse web data toward curated code and mathematical content, optimized with Muon under FP8 hybrid precision and a Warmup-Hold-Decay schedule with linear decay to zero. The pre-trained base is extended to a 128K context window via a layer-selective YaRN and then post-trained in two stages (supervised fine-tuning followed by RLVR), yielding two released variants: an Instruct model that answers directly and a Thinking model that emits an explicit reasoning trace before its final answer. Across code generation, math and reasoning, tool use, knowledge, and safety benchmarks, Mellum 2 is competitive with open-weight baselines in the 4B-14B range while running at the per-token compute of a 2.5B dense model. We release the base, instruct, and thinking checkpoints, together with this report on the architecture decisions, data pipeline, and training recipe behind them, under the Apache 2.0 license.
Alex: Doesn't that mean it loses track of things defined earlier in the file?
Sam: That's a fair concern. Their solution is a hybrid. Most of the model's layers use that fast, local window. But one in every four layers uses full attention—meaning it does look at everything. So you get the speed of the focused approach for most of the work, while a few layers maintain the broader picture.
Alex: And for really long files—like an entire software project—does that still hold up?
Sam: That's where another technique comes in. Every AI model has an internal sense of position—a way of tracking where each word sits relative to everything else. By default, Mellum 2's positioning system was designed for sequences of around 8,000 tokens, which is roughly a few thousand words. To handle much longer files, they apply something called YaRN, which recalibrates that internal sense of distance so the model can track relationships across sequences of up to 128,000 tokens. Critically, they only apply this recalibration to those "global" attention layers—the ones already doing the long-range work. The local layers are left unchanged, which keeps things efficient.
Alex: So the architecture is essentially layered—fast local processing for most tasks, with a few carefully tuned layers handling the big picture.
Sam: Exactly. And that same principle of layered efficiency carries through into how the model is trained. Once the base model is built, they teach it to be a useful assistant in two stages. The first is called Supervised Fine-Tuning. They show the model thousands of examples of high-quality coding conversations and tasks—essentially a curated textbook of good answers—so it learns the expected format and style.
Alex: And the second stage?
Sam: The second stage is where it gets more interesting. Rather than relying on human judges to say whether a response is good—which is slow and subjective—they use what the paper calls Reinforcement Learning with Verifiable Rewards. The idea is to give the model tasks where the answer is objectively checkable. If the model writes code and that code passes a test suite, it gets a reward signal. If it fails, it gets nothing. The model learns to prioritize correctness because it's being graded by a computer sandbox, not a person's opinion.
Alex: So the model is essentially checking its own homework, over and over, against a machine that can't be fooled.
Sam: Right. And because the feedback is unambiguous—the code either works or it doesn't—the model gets a very clean learning signal. To keep this from slowing down training, they separate the grading from the learning. A dedicated cluster of servers runs the code tests, while the training process continues in parallel. The two don't have to wait on each other.
Alex: Every part of this design seems to be solving the same underlying problem—how do you make a system that's both capable enough to be useful and efficient enough to actually run in a real product?
Sam: That's a precise summary of what the paper is trying to demonstrate. They've separated the total knowledge stored in the model from the computational cost of using it at any given moment. The result, according to the paper, is a model that can compete with much larger systems on coding tasks while remaining practical to deploy on standard hardware. Whether that balance holds up across a wider range of real-world conditions is something further work would need to establish—but as a demonstration of the design principles, it's a meaningful piece of engineering.
Alex: Thanks for walking us through it. And thanks to everyone listening to ResearchPod.