LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models | Fengqi Zhu et al. | ResearchPod

Transcript: LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new paper on a type of AI model called a Diffusion Language Model — or dLLM for short.

Sam: Most AI text models work by predicting the next word in a sequence — like dominoes falling one after another. But dLLMs take a completely different approach. Instead of building text forward, they start with a scrambled, corrupted mess of words and gradually clean it up until it makes sense. The challenge is that researchers have been training these models using the same "flight manual" written for those standard, next-word-prediction systems.

Alex: So this paper is asking whether we need a completely different set of instructions for diffusion models?

Sam: Exactly. Because dLLMs learn by fixing errors rather than predicting what comes next, they have different needs — specifically around how much data they see at once, and how quickly they adjust as they learn. If you use the old manual, you're essentially wasting computing power. The researchers developed a new one by systematically testing how these models behave when you give them more resources.

Alex: That makes sense. It's like trying to train a marathon runner on a sprinter's diet — you might get somewhere, but you're not optimizing for the actual task. What specifically did they find was wrong with the old approach?

Sam: Two things stood out. First, as you increase the computing budget, dLLMs need much larger batches of data than previously assumed. Think of a "batch" like a study session — a bigger batch means the model sees more examples at once before updating what it's learned. Second, something called the learning rate — essentially, how boldly the model revises its understanding after each study session — needs to shrink faster than the old guidelines suggested. It's like tuning a high-performance engine. The engine itself hasn't changed, but the fuel mixture and ignition timing need to be dialed in specifically for this type of combustion.

Alex: So the model is the same, but the training recipe is different. You mentioned these are also "Mixture-of-Experts" models — what does that mean?

Sam: A standard AI model uses its entire network for every single word it processes. A Mixture-of-Experts model is more selective. Imagine a hospital where, instead of one general doctor handling every case, you have a whole roster of specialists — a cardiologist, a neurologist, a radiologist — and for each patient, only the relevant two or three are called in. The model works the same way: it has a large library of specialist sub-networks, and for each word, it only activates a small handful. This means the model can be enormous in total capacity, but still fast and efficient in practice.

LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models

Fengqi Zhu, Shaoxuan Xu, Jingyang Ou, Zebin You, Yipeng Xing, Huabin Liu, Xiaolu Zhang, Jun Zhou, Zhenzhong Lan, Yankai Lin, Wayne Xin Zhao, Jianguo Li, Chongxuan Li, Ji-Rong Wen

Aug 4, 2026·5:29·arxiv·Original Paper

Diffusion Language Models (dLLMs)Mixture-of-Experts (MoE)Scaling LawsIsoFLOP AnalysisMoE dLLMsDiffusion Language Models

Listen

5 min

Briefing

Research Approach

This study systematically investigates the scaling behavior of Mixture-of-Experts (MoE) diffusion language models (dLLMs). While autoregressive (AR) models have well-established scaling laws, dLLMs—which use iterative denoising to reconstruct masked tokens—operate under different optimization dynamics. The authors conduct controlled sweeps across compute budgets (10^17 to 10^20 FLOPs) to determine how optimization hyperparameters, compute allocation, and architectural choices (activation ratio, expert granularity, and shared capacity) should scale.

Key Findings

Optimization: Optimal batch sizes for MoE dLLMs grow more steeply with compute than in AR models, while the optimal learning rate decays more rapidly. This is attributed to the reduced effective supervision per token in the masked denoising objective.
Compute Allocation: The optimal strategy favors a slight data-side tilt, where the token budget grows faster than the activated model-side computation. This trend is more pronounced in MoE dLLMs than in dense dLLMs or AR models.
Architecture: Larger scales favor lower activation ratios (increased sparsity). Moderate expert granularity (G=8–16) remains robust across scales, and the optimal fraction of capacity assigned to shared experts (S=33.3%) remains stable.

Validation: LLaDA MoE v2

Guided by these scaling laws, the authors trained LLaDA MoE v2, a 30B-A3B model (30B total parameters, 3B activated per token), on 23.5T tokens. Despite using 35% fewer pretraining tokens than Qwen3, the model achieves competitive performance on knowledge, reasoning, and coding benchmarks. After supervised fine-tuning, it outperforms SDAR Chat on seven of eight reasoning and coding tasks, demonstrating the practical utility of these scaling principles.

Key Terms and Definitions

Diffusion Language Model (dLLM) — A model that defines a text distribution through an iterative denoising process, reconstructing masked tokens from a corrupted sequence.
Activated Non-Embedding FLOPs (M) — A measure of model-side computation that accounts for the sparse activation of experts in an MoE architecture.
Activation Ratio (A) — The fraction of total expert capacity that is activated for each token during the forward pass.
Expert Granularity (G) — A parameter controlling how routed capacity is partitioned into individual experts, balancing routing diversity against per-expert capacity.
Shared-Expert Ratio (S) — The proportion of activated expert capacity assigned to a shared pathway that processes every token.

Abstract

Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling, yet the scaling behavior of Mixture-of-Experts (MoE) dLLMs remains poorly understood. We systematically characterize how optimization hyperparameters, compute allocation, and architecture scale for MoE dLLMs, identifying quantitative differences from scaling trends previously reported for AR models. Specifically, for optimization, the optimal nominal batch size grows faster, while the optimal learning rate decays more rapidly with compute. For model--data allocation, IsoFLOP analysis reveals a slight data-side tilt: the optimal token budget grows faster than activated model-side computation. For MoE architecture, larger scales increasingly favor larger expert pools at fixed activated capacity, while moderate expert granularity remains consistently effective and the preferred fraction of activated capacity assigned to shared experts remains stable across scales. Guided by these findings, we train LLaDA MoE v2, a 30B-A3B dLLM, from scratch on 23.5T tokens. With approximately 65\% as many pretraining tokens as Qwen3, LLaDA MoE v2 approaches Qwen3 on several knowledge, reasoning, and coding benchmarks. After supervised fine-tuning alone, it outperforms SDAR Chat on seven of eight reasoning and coding benchmarks and remains close to Qwen3 on several tasks. These results establish practical scaling laws and design principles for MoE dLLMs.

LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models: Research Paper Podcast