The rapid development of large language models has revolutionized code intelligence in software development. However, the predominance of closed-source models has restricted extensive research and development. To address this, we introduce the DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens. These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling. Our extensive evaluations demonstrate that DeepSeek-Coder not only achieves state-of-the-art performance among open-source code models across multiple benchmarks but also surpasses existing closed-source models like Codex and GPT-3.5. Furthermore, DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use.
Alex: Welcome to another episode of ResearchPod.
Sam: Today, we're looking at how people are trying to make computer programs that help write code, but a big roadblock is that the best ones are locked away by companies. The paper introduces DeepSeek-Coder, a set of open-source models for code tasks, sized from 1.3 billion to 33 billion parameters and trained from scratch on two trillion tokens across 87 programming languages. These models aim to close the gap with closed-source giants like GPT-3.5 by being fully open for anyone to use or build on.
Alex: So the core puzzle here is that closed models perform better, but researchers can't access them—so this work tries to match that power openly?
Sam: Exactly. Developers often edit code in huge projects where files depend on each other, like one file pulling in functions from another. Current open models train on single files, missing those links, so they falter on real-world tasks like suggesting completions across a whole codebase. DeepSeek-Coder tackles this with repository-level pretraining—organizing entire project folders so the model learns those connections.
Alex: Meaning, instead of snippets, it's seeing the full picture of how code files rely on each other?
Sam: Yes. They parse dependencies—like spotting import statements—and use a sorting method to line up files in the order they'd run, preserving the project's natural flow. Think of it like lining up recipe steps where one needs ingredients from earlier ones first. This is the first time that's been done at scale in pretraining, and the paper suggests it boosts cross-file understanding significantly. They pair it with a fill-in-the-middle task, where the model practices completing code gaps given hints from both sides.
Alex: Right, and that helps with practical stuff like autocomplete in an IDE when you're midway through a big edit?
Sam: Precisely. The evidence shows their 33-billion model beats GPT-3.5 on many code benchmarks, while a smaller 6.7-billion one rivals models five times its size—making high performance accessible without proprietary barriers.
Alex: But how exactly do they figure out the right sequence for all those files in a project?
Sam: They start by scanning pairs of files in a repository to spot dependencies, like when one imports functions from another using patterns such as 'import' in Python or 'include' in C. This builds a map showing which files rely on which—like a family tree where parents must come before kids. They count incoming links for each file and repeatedly pick the one with the fewest to start the sequence, updating counts as they go. This handles even messy loops better than strict rules. Researchers call this process a *topological sort*, and it lets them concatenate files in a logical flow, adding path comments so the model knows where everything sits.
Alex: Doesn't that make the training data huge, though?
Sam: It does, so they crawl GitHub for repos in 87 languages, then apply strict filters—like limiting line lengths, checking for enough letters versus symbols, and skipping data-heavy formats such as XML or JSON. This cuts the raw pile down to about a third its size. They also deduplicate at the whole repository level, treating concatenated files as one unit to avoid breaking project structures, which past work shows boosts model quality. Further steps use compilers to catch syntax errors, quality checks for readability, and n-gram matching to scrub test-set leaks—like exact phrases from benchmarks.
Alex: Right, and mixing in some natural language—like GitHub notes or forum posts—helps too?
Sam: Yes, the final mix is 87 percent pure code, 10 percent English tied to coding from Markdown and Stack Exchange, and 3 percent unrelated Chinese text for broader language skills. They extend context to 16,000 tokens to handle long sequences, training via next-token prediction on these ordered chunks. The paper suggests this pipeline creates a more realistic dataset, enabling meaningful gains in cross-file tasks without proprietary data.
Alex: So with that ordered data in place, how do they train the model to actually fill in code gaps across files?
Sam: They use a training trick where they split a piece of code into three parts: the start, the end, and the middle—like giving puzzle edges first to fill the center. Then they shuffle those parts and feed them to the model with special markers, teaching it to recreate the middle when given the start and end first. This builds skill for real coding, where you often need to insert code based on what's before and after. One way does start then end then middle; the other does end then start then middle. They call these PSM for prefix-suffix-middle and SPM for suffix-prefix-middle, settling on half the training using PSM for a good balance.
Alex: But why half and half—not all one way?
Sam: They tested different mixes on a small version of their model with Python code. Full use of that fill-middle task made it great at gaps but weaker at straight-ahead code writing, showing a trade-off. Half PSM worked about twice as well as another method that hides multiple chunks, hitting strong scores on both gap-filling and completion tests. So they picked that rate, applying it file-by-file before packing data together.
Alex: And to handle whole repos, they stretch the model's memory?
Sam: Yes. Normally, these models remember only a short stretch of code at once, like a few pages. They tweak the math that tracks positions—called Rotary Position Embedding—to double or quadruple that, training extra steps on long 16,000-token chunks. This lets it process bigger contexts reliably, key for repository work, though tests show 16K is the sweet spot even if theory allows more.
Alex: Right, so no forgetting midway through a project file chain. Does the architecture help there too?
Sam: It's a standard transformer setup, layers stacking attention to weigh code parts. The big 33B version groups some attention heads for speed, uses fast math tricks, and a custom tokenizer splitting words into 32,000 common chunks. Optimization follows proven steps: gradual learning rates in phases, big batches on GPU clusters. The paper suggests this solid base, plus their data and tasks, drives the performance edge.
Alex: So the pieces fit: ordered data, gap training, longer memory. A clear path to matching closed models openly. Okay, so all those steps pay off in tests. What do the actual coding challenges show?
Sam: They tested on standard puzzles where models get a problem description and partial code, then must complete it correctly on the first try without hints. One set covers Python and six other languages like C++ or Java; another focuses on data science libraries for real workflows. The paper suggests their base 33B model averages around 50% success across those language puzzles—a clear improvement over open rivals like the 34B CodeLlama—and their tuned version tops GPT-3.5 Turbo there too. This ties back to learning full project flows: the model grasps dependencies better, so it generates fitting code across files.
Alex: How about tougher, contest-style problems?
Sam: For recent competition problems—like those from LeetCode—they collected 180 fresh ones with many test cases each. Their tuned 33B hits about 28% success, beating GPT-3.5 and open peers, though still behind GPT-4. Adding step-by-step planning prompts boosts it further on hard logic, as the model outlines before coding. The evidence points to repository training helping with real dependencies in complex tasks.
Alex: So the full-project view shines in practical coding. Beyond isolated problems, how does it show up in tasks needing context across files?
Sam: A key test looks at real codebases where completion requires pulling info from other files—like guessing a function call based on imports elsewhere. They measure exact matches and how close edits are to perfect. Without the repository ordering in training, their 6.7B model drops noticeably on languages like Java and TypeScript, proving that learning file links directly boosts accuracy there. With it, the model outperforms open rivals like CodeLlama by a clear margin.
Alex: Okay, so the dependency map pays off exactly where single-file training fails. What about filling single-line gaps, like in an editor?
Sam: For line-by-line infilling, they compare on Python, Java, JavaScript puzzles needing exact matches. Their small 1.3B base beats bigger ones like 16B StarCoder overall, thanks to cleaner data from the pipeline. Larger sizes scale up steadily—the 33B hits over 80% mean across languages, a solid gain showing quality matters as much as size. The paper recommends the 6.7B for tools, balancing speed and skill.
Alex: Right, code strength carries over. One caution in the paper caught my eye—data leaks?
Sam: Yes, even with fresh LeetCode picks post-training cutoff, contamination can't be ruled out fully, especially higher scores on recent contests. They urge checking that in future uses. Still, the cross-file edge holds firm without leaks.
Alex: Any tweaks post-base training?
Sam: They continue pretraining a 7B from a general language model on two trillion more tokens—mostly code but with math and bilingual text, using plain next-token prediction at shorter 4K context. This boosts natural language and math without FIM, creating DeepSeek-Coder-v1.5. Early signs suggest gains across tasks, building on the base. The v1.5 version holds steady on most coding scores but shows clear gains in math—solving about 45 percent more grade-school problems via code on one test—and natural language understanding.
Alex: Right, so the code focus carries over to better math and language handling.
Sam: Yes. The paper positions this as evidence that top code models need solid general foundations—human instructions often mix everyday words with code specifics. Their full approach, from project ordering to gap training and extra pretraining, lets open models like the 33B base surpass GPT-3.5 on key coding tests and rival much larger rivals. The extended 16,000-token context works reliably for long code, though theory supports more like 64,000—the paper notes real tests cap at 16K for stability. Possible minor data overlap with recent benchmarks like LeetCode remains a caution, urging checks in deployments. Still, the cross-file gains appear robust.
Alex: Balanced view—strong open tools now, with room to scale.
Sam: In sum, DeepSeek-Coder offers a clear recipe for open models to match closed ones: quality project data, smart training tasks, and general roots. It narrows gaps meaningfully, paving the way for accessible coding aids that grasp full codebases. That's the notable contribution here.
Alex: Well put. Thanks for breaking it down, Sam—this makes the logic behind competitive open code models much clearer. Thanks for listening to ResearchPod.