Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y.K. Li, Fuli Luo, Yingfei Xiong, Wenfeng Liang
11 min
Abstract
The rapid development of large language models has revolutionized code intelligence in software development. However, the predominance of closed-source models has restricted extensive research and development. To address this, we introduce the DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens. These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling. Our extensive evaluations demonstrate that DeepSeek-Coder not only achieves state-of-the-art performance among open-source code models across multiple benchmarks but also surpasses existing closed-source models like Codex and GPT-3.5. Furthermore, DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use.
Alex: Doesn't that make the training data huge, though?
Sam: It does, so they crawl GitHub for repos in 87 languages, then apply strict filters—like limiting line lengths, checking for enough letters versus symbols, and skipping data-heavy formats such as XML or JSON. This cuts the raw pile down to about a third its size. They also deduplicate at the whole repository level, treating concatenated files as one unit to avoid breaking project structures, which past work shows boosts model quality. Further steps use compilers to catch syntax errors, quality checks for readability, and n-gram matching to scrub test-set leaks—like exact phrases from benchmarks.
Alex: Right, and mixing in some natural language—like GitHub notes or forum posts—helps too?
Sam: Yes, the final mix is 87 percent pure code, 10 percent English tied to coding from Markdown and Stack Exchange, and 3 percent unrelated Chinese text for broader language skills. They extend context to 16,000 tokens to handle long sequences, training via next-token prediction on these ordered chunks. The paper suggests this pipeline creates a more realistic dataset, enabling meaningful gains in cross-file tasks without proprietary data.
Alex: So with that ordered data in place, how do they train the model to actually fill in code gaps across files?
Sam: They use a training trick where they split a piece of code into three parts: the start, the end, and the middle—like giving puzzle edges first to fill the center. Then they shuffle those parts and feed them to the model with special markers, teaching it to recreate the middle when given the start and end first. This builds skill for real coding, where you often need to insert code based on what's before and after. One way does start then end then middle; the other does end then start then middle. They call these PSM for prefix-suffix-middle and SPM for suffix-prefix-middle, settling on half the training using PSM for a good balance.
Alex: But why half and half—not all one way?
Sam: They tested different mixes on a small version of their model with Python code. Full use of that fill-middle task made it great at gaps but weaker at straight-ahead code writing, showing a trade-off. Half PSM worked about twice as well as another method that hides multiple chunks, hitting strong scores on both gap-filling and completion tests. So they picked that rate, applying it file-by-file before packing data together.
Alex: And to handle whole repos, they stretch the model's memory?
Sam: Yes. Normally, these models remember only a short stretch of code at once, like a few pages. They tweak the math that tracks positions—called Rotary Position Embedding—to double or quadruple that, training extra steps on long 16,000-token chunks. This lets it process bigger contexts reliably, key for repository work, though tests show 16K is the sweet spot even if theory allows more.
Alex: Right, so no forgetting midway through a project file chain. Does the architecture help there too?
Sam: It's a standard transformer setup, layers stacking attention to weigh code parts. The big 33B version groups some attention heads for speed, uses fast math tricks, and a custom tokenizer splitting words into 32,000 common chunks. Optimization follows proven steps: gradual learning rates in phases, big batches on GPU clusters. The paper suggests this solid base, plus their data and tasks, drives the performance edge.
Alex: So the pieces fit: ordered data, gap training, longer memory. A clear path to matching closed models openly. Okay, so all those steps pay off in tests. What do the actual coding challenges show?
Sam: They tested on standard puzzles where models get a problem description and partial code, then must complete it correctly on the first try without hints. One set covers Python and six other languages like C++ or Java; another focuses on data science libraries for real workflows. The paper suggests their base 33B model averages around 50% success across those language puzzles—a clear improvement over open rivals like the 34B CodeLlama—and their tuned version tops GPT-3.5 Turbo there too. This ties back to learning full project flows: the model grasps dependencies better, so it generates fitting code across files.
Alex: How about tougher, contest-style problems?
Sam: For recent competition problems—like those from LeetCode—they collected 180 fresh ones with many test cases each. Their tuned 33B hits about 28% success, beating GPT-3.5 and open peers, though still behind GPT-4. Adding step-by-step planning prompts boosts it further on hard logic, as the model outlines before coding. The evidence points to repository training helping with real dependencies in complex tasks.
Alex: So the full-project view shines in practical coding. Beyond isolated problems, how does it show up in tasks needing context across files?
Sam: A key test looks at real codebases where completion requires pulling info from other files—like guessing a function call based on imports elsewhere. They measure exact matches and how close edits are to perfect. Without the repository ordering in training, their 6.7B model drops noticeably on languages like Java and TypeScript, proving that learning file links directly boosts accuracy there. With it, the model outperforms open rivals like CodeLlama by a clear margin.
Alex: Okay, so the dependency map pays off exactly where single-file training fails. What about filling single-line gaps, like in an editor?
Sam: For line-by-line infilling, they compare on Python, Java, JavaScript puzzles needing exact matches. Their small 1.3B base beats bigger ones like 16B StarCoder overall, thanks to cleaner data from the pipeline. Larger sizes scale up steadily—the 33B hits over 80% mean across languages, a solid gain showing quality matters as much as size. The paper recommends the 6.7B for tools, balancing speed and skill.
Alex: Right, code strength carries over. One caution in the paper caught my eye—data leaks?
Sam: Yes, even with fresh LeetCode picks post-training cutoff, contamination can't be ruled out fully, especially higher scores on recent contests. They urge checking that in future uses. Still, the cross-file edge holds firm without leaks.
Alex: Any tweaks post-base training?
Sam: They continue pretraining a 7B from a general language model on two trillion more tokens—mostly code but with math and bilingual text, using plain next-token prediction at shorter 4K context. This boosts natural language and math without FIM, creating DeepSeek-Coder-v1.5. Early signs suggest gains across tasks, building on the base. The v1.5 version holds steady on most coding scores but shows clear gains in math—solving about 45 percent more grade-school problems via code on one test—and natural language understanding.
Alex: Right, so the code focus carries over to better math and language handling.
Sam: Yes. The paper positions this as evidence that top code models need solid general foundations—human instructions often mix everyday words with code specifics. Their full approach, from project ordering to gap training and extra pretraining, lets open models like the 33B base surpass GPT-3.5 on key coding tests and rival much larger rivals. The extended 16,000-token context works reliably for long code, though theory supports more like 64,000—the paper notes real tests cap at 16K for stability. Possible minor data overlap with recent benchmarks like LeetCode remains a caution, urging checks in deployments. Still, the cross-file gains appear robust.
Alex: Balanced view—strong open tools now, with room to scale.
Sam: In sum, DeepSeek-Coder offers a clear recipe for open models to match closed ones: quality project data, smart training tasks, and general roots. It narrows gaps meaningfully, paving the way for accessible coding aids that grasp full codebases. That's the notable contribution here.
Alex: Well put. Thanks for breaking it down, Sam—this makes the logic behind competitive open code models much clearer. Thanks for listening to ResearchPod.