CASE: Causal Alignment and Structural Enforcement for Improving Chain-of-Thought Faithfulness

Large language models (LLMs) often generate reasoning chains that do not actually determine their final answers, relying instead on direct shortcuts from the input instruction. This paper investigates how to enforce a faithful causal chain (Instruction → Reasoning → Answer) to ensure that the generated reasoning is not just a plausible narrative, but the actual basis for the model's decision.

The authors propose CASE (Causal Alignment and Structural Enforcement), a two-part framework designed to align the model's internal decision-making with the desired causal path.

1. Training-time Causal Alignment: The researchers augment standard training data with three types of synthetic datasets: counterfactual-CoT (changing the reasoning to force a different answer), biased-instruction (adding misleading cues to the instruction), and empty-instruction (removing the instruction entirely). By applying selective-loss fine-tuning, the model learns to prioritize the reasoning chain over the instruction when predicting the final answer.
2. Inference-time Structural Enforcement: During generation, the model applies an attention mask that prevents answer tokens from directly attending to instruction tokens. This forces the model to rely on the hidden states generated during the reasoning phase, effectively severing the direct instruction-to-answer shortcut in the computation graph.

CASE demonstrates significant improvements in faithfulness across multiple benchmarks and models. Compared to the strongest baselines, it achieves an average 37% relative improvement in faithfulness metrics while maintaining competitive accuracy. The authors provide an information-theoretic analysis demonstrating that their training objectives effectively increase the mutual information between the reasoning and the answer, while minimizing the residual dependence of the answer on the instruction.