Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time. We study this setting as document knowledge internalization: converting a fixed corpus into usable parametric knowledge for retrieval-free question answering. We propose IAR (Inject, Align, and Recover), a three-stage post-training framework that separates structured document knowledge injection, QA behavior alignment, and general ability recovery. Unlike conventional continued pretraining, Inject converts source documents into continuation, rewrite, and instruction-conditioned reconstruction objectives. Align then adapts the injected model with answer-only QA supervision, while Recover merges the domain-adapted model with the base instruction model to recover general capabilities. Across Common Corpus (CC) and CCI, and across Llama, Phi, Qwen, and SmolLM model families, IAR improves the domain-primary domain-general frontier for retrieval-free document internalization. In the main comparison, IAR improves over Vanilla SFT on all four reported metrics in 7 of 8 dataset-model settings, with average gains of 3.6 percentage points in domain QA accuracy and 12.1 percentage points in mean general performance across IFEval, MMLU, and MSBench. Extended CC baselines show that LoRA and FAPM can win individual general metrics, but among methods that also reach leading or near-leading domain internalization, IAR retains one of the strongest general profiles.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at a paper by Qian Kou and colleagues from the Beijing Academy of Artificial Intelligence. The question they're asking is deceptively simple: how do you get a language model to truly know a private document collection — not just look things up in it, but have that knowledge baked in?
Alex: That's an interesting distinction. Most systems I've heard of just search through documents when you ask a question. Why would you want to do it differently?
Sam: Right, so the standard approach is called retrieval-augmented generation. Think of it like an open-book exam — the model gets to flip through a stack of documents every time it answers a question. That works well when you have access to those documents at the moment of answering. But sometimes you don't. Maybe the documents are confidential, or the system needs to work offline, or retrieval is just too slow. In those cases, the model has to answer from memory alone.
Alex: So you need the knowledge permanently stored inside the model itself.
Sam: Exactly. The authors call this "document knowledge internalization." And the challenge is that the obvious training approaches all have serious drawbacks. If you train the model intensively on your private documents, it gets good at those — but it starts forgetting how to do everything else. It's like studying one subject so hard before an exam that you blank on all the others.
Alex: That forgetting problem sounds like the central obstacle here.
Sam: It is. There's a specific name for it — catastrophic forgetting — and it's a well-known issue in machine learning. When you update a model's internal settings to learn new information, those updates can overwrite the patterns it already had. So the researchers needed a way to teach the model new domain knowledge without erasing its general reasoning ability.
Alex: How do they approach that?
Sam: They design a three-stage framework they call IAR — Inject, Align, and Recover. Think of it like training a student to become a specialist. First, you have them deeply reread and reconstruct the source material. Then you run them through practice questions so they learn how to apply that knowledge in conversation. And finally, you carefully blend their new specialist knowledge back with their general education so they don't come out the other side unable to do basic reasoning.
Alex: Walk me through those stages. What's actually happening in each one?
Sam: In the first stage, Inject, the model is trained on the documents themselves — but not just by reading them passively. It's given reconstruction tasks, essentially being asked to reproduce and engage with the content deeply. The goal is dense exposure to the material. The second stage, Align, shifts the format. Now the model is trained on question-and-answer pairs drawn from those documents, so it learns to surface that knowledge in a conversational way.
Alex: And the third stage is where the forgetting gets addressed?
Sam: That's the Recover stage, and it's arguably the most technically interesting part. By the time you've done the first two stages, the model is good at your documents — but its general capabilities have taken a hit. So the researchers combine the internal settings of the specialized model with those of the original, general-purpose model.
Alex: What do you mean by internal settings exactly?
Sam: Think of them as millions of tiny dials inside the system — each one tuned to shape how the model thinks and responds. After fine-tuning on your documents, some of those dials have been turned in ways that conflict with the general knowledge the model had before. The merging process identifies which adjustments are genuinely important for the new domain knowledge, trims the redundant ones, and resolves cases where the two models are pulling in opposite directions. The researchers use a technique called TIES-Merging for this — the name isn't critical, but the idea is that you're selectively blending two expert brains by keeping only the most meaningful changes from each.
Alex: And does it actually work? Does the general capability come back?
Sam: The results are meaningful. The recovery stage restores a substantial portion of general reasoning and instruction-following ability — while domain accuracy drops by only a very small margin. So you're trading a tiny fraction of specialist performance to recover most of the general capability you'd lost.
Alex: That's a much better trade than just picking one or the other.
Sam: That's precisely the point. An organization doesn't have to choose between a model that knows their proprietary documents and one that can still follow basic instructions or reason through novel problems. The framework treats those as separable concerns.
Alex: How do the researchers decide which specific merging settings to use? There must be a lot of choices involved.
Sam: There are. They test several different merging approaches across a range of configurations, and they select the best one using an internal validation set — not the final test data. That's an important methodological detail, because it means the selection process isn't accidentally tuned to the test results. They filter first for models that retain strong domain accuracy, then check that general capabilities have recovered above a minimum threshold.
Alex: Is there a universal recipe that works across all models and document types?
Sam: No, and the paper is candid about that. The best settings depend heavily on the specific model and the specific document collection. For instance, some models already have a strong prior understanding of certain types of content — like code or structured text — so the training has to be more careful not to disturb what's already there. The recovery stage becomes especially valuable in those cases.
Alex: What about simpler alternatives — like just cutting out parts of the fine-tuning that caused the most damage?
Sam: The researchers compare against approaches that selectively remove parameter updates rather than merging two models together. Those methods can improve general benchmark scores, but they tend to damage the newly acquired domain knowledge in the process. The weight-space merging approach avoids that particular trade-off more cleanly.
Alex: So the broader takeaway is that you can update a model's permanent memory in a structured, recoverable way — rather than it being a one-way door.
Sam: That's a good way to put it. By separating the process into ingestion, alignment, and recovery, domain adaptation becomes modular. Each stage has a clear purpose, and the damage done in earlier stages can be meaningfully repaired in later ones. The implication is that future systems might be able to absorb entire private document libraries without needing external search infrastructure every time someone asks a question — and without losing the general reasoning that makes them useful in the first place.
Alex: That's a meaningful shift in how we think about what a model can permanently know. Thanks for walking us through it, and thanks to everyone for listening to ResearchPod.