Qian Kou, Xiaofeng Shi, Xiaosong Qiu, Hua Zhou
6 min
Abstract
Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time. We study this setting as document knowledge internalization: converting a fixed corpus into usable parametric knowledge for retrieval-free question answering. We propose IAR (Inject, Align, and Recover), a three-stage post-training framework that separates structured document knowledge injection, QA behavior alignment, and general ability recovery. Unlike conventional continued pretraining, Inject converts source documents into continuation, rewrite, and instruction-conditioned reconstruction objectives. Align then adapts the injected model with answer-only QA supervision, while Recover merges the domain-adapted model with the base instruction model to recover general capabilities. Across Common Corpus (CC) and CCI, and across Llama, Phi, Qwen, and SmolLM model families, IAR improves the domain-primary domain-general frontier for retrieval-free document internalization. In the main comparison, IAR improves over Vanilla SFT on all four reported metrics in 7 of 8 dataset-model settings, with average gains of 3.6 percentage points in domain QA accuracy and 12.1 percentage points in mean general performance across IFEval, MMLU, and MSBench. Extended CC baselines show that LoRA and FAPM can win individual general metrics, but among methods that also reach leading or near-leading domain internalization, IAR retains one of the strongest general profiles.
Alex: And the third stage is where the forgetting gets addressed?
Sam: That's the Recover stage, and it's arguably the most technically interesting part. By the time you've done the first two stages, the model is good at your documents — but its general capabilities have taken a hit. So the researchers combine the internal settings of the specialized model with those of the original, general-purpose model.
Alex: What do you mean by internal settings exactly?
Sam: Think of them as millions of tiny dials inside the system — each one tuned to shape how the model thinks and responds. After fine-tuning on your documents, some of those dials have been turned in ways that conflict with the general knowledge the model had before. The merging process identifies which adjustments are genuinely important for the new domain knowledge, trims the redundant ones, and resolves cases where the two models are pulling in opposite directions. The researchers use a technique called TIES-Merging for this — the name isn't critical, but the idea is that you're selectively blending two expert brains by keeping only the most meaningful changes from each.
Alex: And does it actually work? Does the general capability come back?
Sam: The results are meaningful. The recovery stage restores a substantial portion of general reasoning and instruction-following ability — while domain accuracy drops by only a very small margin. So you're trading a tiny fraction of specialist performance to recover most of the general capability you'd lost.
Alex: That's a much better trade than just picking one or the other.
Sam: That's precisely the point. An organization doesn't have to choose between a model that knows their proprietary documents and one that can still follow basic instructions or reason through novel problems. The framework treats those as separable concerns.
Alex: How do the researchers decide which specific merging settings to use? There must be a lot of choices involved.
Sam: There are. They test several different merging approaches across a range of configurations, and they select the best one using an internal validation set — not the final test data. That's an important methodological detail, because it means the selection process isn't accidentally tuned to the test results. They filter first for models that retain strong domain accuracy, then check that general capabilities have recovered above a minimum threshold.
Alex: Is there a universal recipe that works across all models and document types?
Sam: No, and the paper is candid about that. The best settings depend heavily on the specific model and the specific document collection. For instance, some models already have a strong prior understanding of certain types of content — like code or structured text — so the training has to be more careful not to disturb what's already there. The recovery stage becomes especially valuable in those cases.
Alex: What about simpler alternatives — like just cutting out parts of the fine-tuning that caused the most damage?
Sam: The researchers compare against approaches that selectively remove parameter updates rather than merging two models together. Those methods can improve general benchmark scores, but they tend to damage the newly acquired domain knowledge in the process. The weight-space merging approach avoids that particular trade-off more cleanly.
Alex: So the broader takeaway is that you can update a model's permanent memory in a structured, recoverable way — rather than it being a one-way door.
Sam: That's a good way to put it. By separating the process into ingestion, alignment, and recovery, domain adaptation becomes modular. Each stage has a clear purpose, and the damage done in earlier stages can be meaningfully repaired in later ones. The implication is that future systems might be able to absorb entire private document libraries without needing external search infrastructure every time someone asks a question — and without losing the general reasoning that makes them useful in the first place.
Alex: That's a meaningful shift in how we think about what a model can permanently know. Thanks for walking us through it, and thanks to everyone for listening to ResearchPod.