Qian Kou, Xiaofeng Shi, Xiaosong Qiu, Hua Zhou
6 min
When large language models must answer questions about a bounded document collection without retrieving source documents at inference time, they often fail. This paper studies this setting as document knowledge internalization, which converts a fixed corpus into usable parametric knowledge. The authors investigate whether standard supervised fine-tuning and continued pretraining force a damaging trade-off between acquiring domain facts and retaining general model capabilities.
To address this, the authors propose IAR (Inject, Align, and Recover), a three-stage post-training framework that separates these functions. The first stage, Inject, converts source documents into structured continuation, rewrite, and instruction-conditioned reconstruction objectives instead of raw text streaming. The second stage, Align, adapts the model using answer-only question-answering supervision. The third stage, Recover, uses post-hoc model merging between the domain-adapted checkpoint and the original instruction model to restore general capabilities under a validated frontier selection rule.
Across the Common Corpus (CC) and CCI datasets, as well as Llama, Phi, Qwen, and SmolLM model families, IAR successfully improves the domain-general operating frontier. In the main comparisons, IAR improves over Vanilla SFT on all four reported metrics (domain QA accuracy, IFEval, MMLU, and MSBench) in seven of eight dataset-model settings, achieving substantial gains in both domain accuracy and general capability retention.
Token-budget matched experiments confirm that these improvements stem from the structured multi-stage framework rather than simple training length extensions. While extended baseline stress tests show that alternative methods like LoRA or FAPM can occasionally win individual general metrics, IAR maintains one of the strongest overall general profiles among methods reaching leading domain internalization.
These findings suggest that document exposure, answer alignment, data recipes, token budgets, and capability recovery should be decoupled during post-training rather than merged into a single training step. By treating model merging as an explicit recovery phase, practitioners can better balance the tension between memorizing target document collections and preserving general instruction-following competencies.
Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time. We study this setting as document knowledge internalization: converting a fixed corpus into usable parametric knowledge for retrieval-free question answering. We propose IAR (Inject, Align, and Recover), a three-stage post-training framework that separates structured document knowledge injection, QA behavior alignment, and general ability recovery. Unlike conventional continued pretraining, Inject converts source documents into continuation, rewrite, and instruction-conditioned reconstruction objectives. Align then adapts the injected model with answer-only QA supervision, while Recover merges the domain-adapted model with the base instruction model to recover general capabilities. Across Common Corpus (CC) and CCI, and across Llama, Phi, Qwen, and SmolLM model families, IAR improves the domain-primary domain-general frontier for retrieval-free document internalization. In the main comparison, IAR improves over Vanilla SFT on all four reported metrics in 7 of 8 dataset-model settings, with average gains of 3.6 percentage points in domain QA accuracy and 12.1 percentage points in mean general performance across IFEval, MMLU, and MSBench. Extended CC baselines show that LoRA and FAPM can win individual general metrics, but among methods that also reach leading or near-leading domain internalization, IAR retains one of the strongest general profiles.
Alex: And the third stage is where the forgetting gets addressed?
Sam: That's the Recover stage, and it's arguably the most technically interesting part. By the time you've done the first two stages, the model is good at your documents — but its general capabilities have taken a hit. So the researchers combine the internal settings of the specialized model with those of the original, general-purpose model.
Alex: What do you mean by internal settings exactly?
Sam: Think of them as millions of tiny dials inside the system — each one tuned to shape how the model thinks and responds. After fine-tuning on your documents, some of those dials have been turned in ways that conflict with the general knowledge the model had before. The merging process identifies which adjustments are genuinely important for the new domain knowledge, trims the redundant ones, and resolves cases where the two models are pulling in opposite directions. The researchers use a technique called TIES-Merging for this — the name isn't critical, but the idea is that you're selectively blending two expert brains by keeping only the most meaningful changes from each.
Alex: And does it actually work? Does the general capability come back?
Sam: The results are meaningful. The recovery stage restores a substantial portion of general reasoning and instruction-following ability — while domain accuracy drops by only a very small margin. So you're trading a tiny fraction of specialist performance to recover most of the general capability you'd lost.
Alex: That's a much better trade than just picking one or the other.
Sam: That's precisely the point. An organization doesn't have to choose between a model that knows their proprietary documents and one that can still follow basic instructions or reason through novel problems. The framework treats those as separable concerns.
Alex: How do the researchers decide which specific merging settings to use? There must be a lot of choices involved.
Sam: There are. They test several different merging approaches across a range of configurations, and they select the best one using an internal validation set — not the final test data. That's an important methodological detail, because it means the selection process isn't accidentally tuned to the test results. They filter first for models that retain strong domain accuracy, then check that general capabilities have recovered above a minimum threshold.
Alex: Is there a universal recipe that works across all models and document types?
Sam: No, and the paper is candid about that. The best settings depend heavily on the specific model and the specific document collection. For instance, some models already have a strong prior understanding of certain types of content — like code or structured text — so the training has to be more careful not to disturb what's already there. The recovery stage becomes especially valuable in those cases.
Alex: What about simpler alternatives — like just cutting out parts of the fine-tuning that caused the most damage?
Sam: The researchers compare against approaches that selectively remove parameter updates rather than merging two models together. Those methods can improve general benchmark scores, but they tend to damage the newly acquired domain knowledge in the process. The weight-space merging approach avoids that particular trade-off more cleanly.
Alex: So the broader takeaway is that you can update a model's permanent memory in a structured, recoverable way — rather than it being a one-way door.
Sam: That's a good way to put it. By separating the process into ingestion, alignment, and recovery, domain adaptation becomes modular. Each stage has a clear purpose, and the damage done in earlier stages can be meaningfully repaired in later ones. The implication is that future systems might be able to absorb entire private document libraries without needing external search infrastructure every time someone asks a question — and without losing the general reasoning that makes them useful in the first place.
Alex: That's a meaningful shift in how we think about what a model can permanently know. Thanks for walking us through it, and thanks to everyone for listening to ResearchPod.