Perfect Detection, Failed Control: The Geometry of Knowing vs. Steering in Language Models

Mechanistic interpretability often operates on the assumption that if we can identify a direction in a model's activations that detects a specific behavior, we can use that same direction to control or steer that behavior. This paper tests this premise by measuring the geometric angle between detection and intervention directions across several language models, including Gemma 2-2B-it.

The researchers contrast two types of behaviors. For output formatting (e.g., markdown vs. JSON), detection and intervention collapse onto a single axis; the same direction that identifies the format also controls it. However, for hallucination—where the model fabricates information about non-existent entities—the relationship breaks down. The model exhibits perfect linear separability (AUC = 1.000) for fake entities, yet the direction carrying this detection signal is nearly perpendicular to the direction that triggers a refusal. This "detection-intervention gap" is not a construction artifact; it persists regardless of whether the detection direction is derived from activations or hand-picked from the output vocabulary.

This finding challenges the common heuristic that "detection is control." The authors demonstrate that the cosine between detection and intervention directions is a consistent, weight-computable signature of this dissociation across different model families and scales. Crucially, they show that this cosine is not a reliable predictor of how steerable a behavior is. Because detection is a high-dimensional class rather than a single vector, steerability is a functional property that cannot be read from a static geometric angle. The study suggests that knowing and steering are separate faculties within the model, and simply identifying a representation of a concept does not grant the researcher a functional 'knob' to manipulate it.