ResearchPod Summary
This paper addresses the critical gap between an LLM's plausible-sounding reasoning and its actual internal decision-making process. While existing methods often focus on inference-time prompting or aligning models with external 'gold' labels, these approaches fail to ensure that the model's self-explanation is causally linked to its internal logic. The authors propose a reinforcement learning (RL) framework that uses counterfactual interventions to create a training signal. By inserting specific factors (such as random words or user-bias phrases) into prompts, the researchers identify which factors actually change the model's decision. They then use an RL objective to reward the model when its explanation correctly mentions these influential factors and ignores non-influential ones.
The researchers find that while supervised fine-tuning can help a model identify influential factors, it does not necessarily lead the model to verbalize those factors in its explanations. In contrast, their RL-based training successfully aligns the model's internal decision drivers with its verbal output. Using the Phi-CCT metric—which measures the correlation between a model's internal decision drivers and its explanations—the authors show that RL-tuned Llama3.1-8B and Qwen3-8B models achieve significant improvements, with scores rising from near-zero to approximately 0.66–0.69. Furthermore, the models demonstrate effective generalization to held-out tasks like StrategyQA without resorting to common reward-hacking behaviors like verbatim copying or degenerate output shortening.
As LLMs are increasingly deployed in high-stakes domains such as law, medicine, and security, the ability to audit their reasoning is paramount. Current models often generate 'plausible' but unfaithful justifications, which can mislead human oversight. By providing a scalable, parameter-level training method to enforce explanatory faithfulness, this work offers a path toward more transparent and trustworthy AI systems that can be held accountable for the actual logic behind their outputs.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.