Can LLMs Reliably Self-Report Adversarial Prefills, and How?

This paper investigates whether large language models (LLMs) possess the introspective capability to recognize when their prior responses were forced by an adversarial prefill attack. While previous research has shown that LLMs can introspect on benign tasks, it remains unclear if this extends to safety contexts where models are coerced into harmful behavior. The authors evaluate ten instruction-tuned models (3B to 70B parameters) across four safety benchmarks, using two distinct follow-up probes: one asking about internal intention ("Did you mean to say that?") and another about external tampering ("Did anyone tamper with your response?").

The researchers find that no model reliably recognizes its own compromised outputs. On average, models claim intent for 27.3% of their prefilled responses, indicating a significant failure in self-monitoring. The authors discover that this introspective signal is primarily a byproduct of the model's internal refusal mechanisms. By orthogonalizing model weights against the identified "refusal direction," the recognition gap between natural and prefilled outputs collapses to near zero. Furthermore, the study demonstrates that introspection is highly probe-dependent; the same model may provide contradictory answers depending on whether the question is framed as an internal mistake or external tampering.

The authors test whether targeted fine-tuning (SFT, GRPO, DPO) can improve introspection. While these methods successfully widen the intention-probe gap on models from 8B to 27B, they fail to improve performance on the tampering probe. Counterintuitively, these interventions often increase the attack success rate, suggesting that attempting to "train" introspection can inadvertently weaken safety alignment. These results highlight a fundamental risk: current LLMs lack a robust sense of self-awareness in adversarial settings, and existing safety mechanisms are insufficient for reliable self-reporting.