DeepSeek-AI
9 min
Abstract
We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models.
Alex: Okay, so instead of hoarding every detail, it keeps a compact version that still lets it reconstruct what's needed. But how does it handle positions, like the order of words?
Sam: Position matters a lot—the model needs to know if a word came early or late in the text, using something like a rotating code embedded in the keys and queries. The problem is, that code doesn't mix well with the squeezing, so they'd have to rebuild everything from scratch each time, wasting time. Their fix is to split it: use extra slim queries per head and one shared position key, both coded separately, then combine them just for the math. This lets the squeezes absorb neatly into other steps for smooth, fast use.
Alex: Huh—so the cache ends up tiny, like what a grouped setup with just a couple groups would need, but it performs even better than the full multi-head version?
Sam: Exactly. The paper shows their approach needs a KV cache roughly equal to a grouped-query setup with only about two groups, yet it matches or beats the full multi-head attention in strength. This means handling much longer texts without crashing, with generation speeds about five times higher than before despite fewer active parts.
Alex: And tying back to the experts, that routing keeps training balanced across devices too?
Sam: Yes—the expert routing picks a top few per token but limits them to fit device setups, avoiding overload from too many fine specialists scattering across machines. Shared experts help fill gaps without redundancy, making the whole training about 42 percent cheaper than a dense model half its effective size, while hitting top scores on reasoning tests.
Alex: So those balance tweaks in routing—like limiting to three devices and shared experts—sound key to keeping training smooth across machines. But how do they prevent some experts from getting ignored or machines from bottlenecking?
Sam: They add extra checks during training to spread the work evenly. First, an expert-level check makes sure each specialist gets roughly the same number of tokens to practice on, like dividing homework fairly among study group members so no one slacks. If some get too few, it pulls the model's overall score down, so they penalize uneven picks. A device-level check does the same for groups of experts on each machine, and a communication check evens out data swaps between machines to avoid traffic jams.
Alex: Okay, that keeps things from collapsing. But even with that, loads aren't perfect—what's the backup?
Sam: They use a token-dropping trick: on each device, if too many tokens pile up beyond its budget, it skips the ones with weakest matches to available experts, like a restaurant turning away low-priority orders during rush hour. They protect about 10 percent of sequences from any drops, ensuring full training for some full examples.
Alex: Makes sense for efficiency. Now, with all that in place, what did the actual training look like—data scale and setup?
Sam: They pre-trained on 8.1 trillion tokens from cleaned internet sources, with about 12 percent more Chinese than English ones for better bilingual balance. The model has 60 main processing layers; each MoE layer picks 6 routed experts out of 160 plus 2 always-on shared ones, totaling 236 billion parameters but activating just 21 billion per token. After base training, they stretched position tracking with a method called YaRN to cover up to 128 thousand tokens without forgetting order. Tests showed strong retrieval of facts from haystack-long texts.
Alex: And handling super-long texts later?
Sam: The paper suggests this yields reliable long-context handling with the slim cache.
Alex: That long-context stretch seems solid for handling big inputs. But after all the base training, how do they turn it into something useful for chats—like making sure it follows instructions without going off track?
Sam: They start with supervised fine-tuning on about 1.5 million example conversations—mostly focused on being helpful, with some on staying safe—like practicing scripted dialogues to learn polite, accurate replies instead of rambling. Then they follow with reinforcement learning: the model generates multiple replies to the same prompt, scores them based on quality—like how correct or safe they are—and tweaks itself to favor the better ones, without needing a separate judge model.
Alex: Okay, so it learns to chat helpfully from examples and self-improves in batches. And the payoff in actual use?
Sam: The aligned version hits top scores among open-source models on reasoning, math, code, and bilingual tests—outperforming their prior dense 67-billion model across nearly all, despite activating just 21 billion parameters. It matches or beats models like a 72-billion dense one on most English and math tasks, and leads on Chinese ones. Inference runs about six times faster than that prior model on standard hardware.
Alex: So fewer active parts, smarter alignment, and real speed gains—makes sense why it's practical to deploy. Ties the efficiency back to everyday chats without the crashes.
Sam: Precisely. The paper suggests these steps unlock strong, balanced performance at lower cost, a notable advance for accessible large models.
Alex: Overall, these refinements deliver solid results across tests. What stands out most in the comparisons to other models?
Sam: It outperforms a 72-billion dense model from Qwen on most English, math, and code benchmarks, while matching a 70-billion one like LLaMA3 on those areas—though it trails slightly on some commonsense tests. The reinforcement learning step pushes open-ended chats even higher, leading on English conversation scores and topping Chinese leaderboards among open models. They note an alignment tax—refining for helpful chats can dip scores on certain reasoning benchmarks—but they eased it with better data.
Alex: Right—strong bilingual edge, but with some trade-offs. What about broader limits?
Sam: The model shares common issues with others: no real-time knowledge after training, risks of wrong facts or made-up details, and less skill in languages beyond Chinese and English. English commonsense lags a bit versus top dense models. Still, with the cache cut by over 93 percent and training 42 percent cheaper, it runs about six times faster—making 128-thousand-token chats feasible on everyday hardware.
Alex: That efficiency could mean bigger models on consumer devices without huge power needs. Ties back to why the architecture matters.
Sam: Exactly—a meaningful step for open models, strongest among MoE ones with just 21 billion active parameters.
Alex: Thanks, Sam—clear picture of careful progress. Thanks for joining us on ResearchPod.