Knowledge Knows, Verbalization Tells: Disentangling Latent Directions for Mathematical Solvability in LLMs

This paper investigates why Large Language Models (LLMs) often fail to recognize unsolvable mathematical problems, a phenomenon known as fabrication. While previous research has explored internal solvability beliefs, this study distinguishes between two internal representations: knowledge (whether the model internally recognizes a problem as unsolvable) and verbalization (whether the model explicitly communicates this judgment). The authors use linear probing to isolate these two directions within the hidden states of several LLMs, including both instruction-tuned and reasoning-oriented models. They then analyze how these representations interact with prompting and whether they can be mechanistically manipulated to improve abstention.

• Distinct Representations: The authors demonstrate that knowledge and verbalization are encoded as separate, linearly decodable directions in the model's hidden states. Verbalization is consistently easier to decode than knowledge across all tested models.
• Fabrication Dynamics: Fabrication is primarily associated with a misalignment between these two representations. When models fail to abstain on unsolvable problems, it is usually because the verbalization direction shifts away from the knowledge direction, rather than a failure in the underlying knowledge itself.
• Prompting Effects: Prompting with unsolvability cues reduces fabrication by primarily shifting the verbalization representation. In reasoning models, the internal thinking trace and the final output trace show distinct dynamics, with verbalization consolidating gradually during the generation process.
• Mechanistic Steering: The authors introduce a gated activation steering technique that selectively intervenes in the model's hidden states. They find that jointly steering both the knowledge and verbalization directions (KV steering) is more effective at rescuing abstention than steering either direction alone, provided the intervention is gated to prevent false positives on solvable problems.

This work provides a mechanistic explanation for why LLMs often "know" a problem is unsolvable but still choose to fabricate an answer. By disentangling internal knowledge from verbal output, the authors offer a framework for improving model reliability. The success of gated steering suggests that we can improve model honesty without needing to retrain the model, simply by realigning its internal verbalization with its latent knowledge.