Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPO

Language models often prioritize their internal, memorized knowledge over evidence provided in a prompt, a failure known as a lack of context grounding. While various post-training techniques—such as Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO)—are used to improve this behavior, it remains unclear whether these methods build new internal structures or simply repurpose and strengthen mechanisms already present in the model. The authors conduct a comprehensive audit of nine different post-training recipes starting from the same instruction-tuned checkpoint, using causal analysis and activation steering to determine how these models change internally.

The study reveals a sharp dissociation between training recipes. GRPO variants yield only small, often non-robust gains in context grounding, even when the model successfully improves on the specific metrics it is rewarded for. In contrast, conflict-aware SFT provides moderate improvements, and DPO achieves near-ceiling grounding performance across multiple model families and datasets.

Crucially, the authors find that the gains from SFT and DPO rely on pre-existing machinery. By identifying a grounding direction in the starting model before any training occurs, the researchers demonstrate that this direction is causally linked to the model's behavior. Subtracting this direction from trained models significantly suppresses their grounding gains, while adding it to the starting model increases its grounding performance. Furthermore, mechanistic analysis shows that these training methods largely recover the same set of causal attention heads identified in the starting model, suggesting that post-training is primarily an act of amplification rather than architectural innovation.

This work provides a mechanistic account of how alignment techniques shape model behavior. By showing that grounding gains are often the result of recruiting existing capabilities, the study suggests that the effectiveness of post-training is constrained by the starting model's internal structure. This has significant implications for researchers aiming to improve model faithfulness, as it highlights that the "raw material" present in a base model may be a primary determinant of the success of subsequent alignment efforts.