arxiv

LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models

Fengqi Zhu, Shaoxuan Xu, Jingyang Ou, Zebin You et al.Aug 4, 2026Original

ResearchPod Summary

Research Approach

This study systematically investigates the scaling behavior of Mixture-of-Experts (MoE) diffusion language models (dLLMs). While autoregressive (AR) models have well-established scaling laws, dLLMs—which use iterative denoising to reconstruct masked tokens—operate under different optimization dynamics. The authors conduct controlled sweeps across compute budgets (10^17 to 10^20 FLOPs) to determine how optimization hyperparameters, compute allocation, and architectural choices (activation ratio, expert granularity, and shared capacity) should scale.

Key Findings

Optimization: Optimal batch sizes for MoE dLLMs grow more steeply with compute than in AR models, while the optimal learning rate decays more rapidly. This is attributed to the reduced effective supervision per token in the masked denoising objective.
Compute Allocation: The optimal strategy favors a slight data-side tilt, where the token budget grows faster than the activated model-side computation. This trend is more pronounced in MoE dLLMs than in dense dLLMs or AR models.
Architecture: Larger scales favor lower activation ratios (increased sparsity). Moderate expert granularity (G=8–16) remains robust across scales, and the optimal fraction of capacity assigned to shared experts (S=33.3%) remains stable.

Validation: LLaDA MoE v2

Guided by these scaling laws, the authors trained LLaDA MoE v2, a 30B-A3B model (30B total parameters, 3B activated per token), on 23.5T tokens. Despite using 35% fewer pretraining tokens than Qwen3, the model achieves competitive performance on knowledge, reasoning, and coding benchmarks. After supervised fine-tuning, it outperforms SDAR Chat on seven of eight reasoning and coding tasks, demonstrating the practical utility of these scaling principles.

Key Terms

Paper Details

arXiv: 2608.03457

Topics

Diffusion Language Models (dLLMs)Mixture-of-Experts (MoE)Scaling LawsIsoFLOP AnalysisMoE dLLMsDiffusion Language ModelsCompute AllocationExpert GranularityMixture-of-Experts (MoE) TransformerdLLMs (Denoising Large Language Models)Compute-Dependent HyperparametersdLLM (denoising LLM)

Podcast transcript

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new paper on a type of AI model called a Diffusion Language Model — or dLLM for short.

Sam: Most AI text models work by predicting the next word in a sequence — like dominoes falling one after another. But dLLMs take a completely different approach. Instead of building text forward, they start with a scrambled, corrupted mess of words and gradually clean it up until it makes sense. The challenge is that researchers have been training these models using the same "flight manual" written for those standard, next-word-prediction systems.

Alex: So this paper is asking whether we need a completely different set of instructions for diffusion models?

Sam: Exactly. Because dLLMs learn by fixing errors rather than predicting what comes next, they have different needs — specifically around how much data they see at once, and how quickly they adjust as they learn. If you use the old manual, you're essentially wasting computing power. The researchers developed a new one by systematically testing how these models behave when you give them more resources.

Alex: That makes sense. It's like trying to train a marathon runner on a sprinter's diet — you might get somewhere, but you're not optimizing for the actual task. What specifically did they find was wrong with the old approach?

Sam: Two things stood out. First, as you increase the computing budget, dLLMs need much larger batches of data than previously assumed. Think of a "batch" like a study session — a bigger batch means the model sees more examples at once before updating what it's learned. Second, something called the learning rate — essentially, how boldly the model revises its understanding after each study session — needs to shrink faster than the old guidelines suggested. It's like tuning a high-performance engine. The engine itself hasn't changed, but the fuel mixture and ignition timing need to be dialed in specifically for this type of combustion.

Alex: So the model is the same, but the training recipe is different. You mentioned these are also "Mixture-of-Experts" models — what does that mean?

Sam: A standard AI model uses its entire network for every single word it processes. A Mixture-of-Experts model is more selective. Imagine a hospital where, instead of one general doctor handling every case, you have a whole roster of specialists — a cardiologist, a neurologist, a radiologist — and for each patient, only the relevant two or three are called in. The model works the same way: it has a large library of specialist sub-networks, and for each word, it only activates a small handful. This means the model can be enormous in total capacity, but still fast and efficient in practice.