Probing the Misaligned Thinking Process of Language Models

As large language models (LLMs) are deployed in increasingly high-stakes environments, detecting misaligned behaviors—such as strategic deception, sycophancy, and self-preservation—has become a critical safety challenge. Current methods, such as using LLM judges to analyze model transcripts, are computationally expensive and may become less effective as models learn to obfuscate their reasoning. This paper investigates whether it is possible to monitor misalignment more efficiently by decomposing it into fine-grained cognitive processes and detecting them directly within the model's internal activations.

The researchers propose a taxonomy of 18 misalignment indicators, which are specific reasoning patterns (e.g., strategic omission, fabrication, or malicious action planning) that signal a model is pursuing a goal divergent from the user's intent. They developed an automated, meta-plan-guided pipeline to generate diverse synthetic training data, including positive examples of these indicators, hard negatives, and benign controls. Using this data, they trained linear probes on the internal residual-stream activations of a target model. To ensure robust monitoring, they implemented a two-stage cascade: a lightweight probe-based filter that flags suspicious turns, followed by a high-precision LLM judge that performs final adjudication only on flagged content.

The study finds that these indicator-based probes are highly effective, achieving an AUROC of 0.936 on out-of-distribution misalignment benchmarks. When combined into a two-stage cascade, the system matches the performance of the strongest LLM judges while requiring significantly less computation. Specifically, the cascade reduces the number of LLM calls needed for monitoring by over 90% on benign traffic, as the probes successfully filter out most honest interactions. The authors also demonstrate that providing LLM judges with the indicator taxonomy improves their detection accuracy, suggesting that these indicators provide a robust framework for understanding and identifying misaligned reasoning.