Protective Capacity Hallucination: When Large Language Models Claim Nonexistent Capabilities

Understanding Protective Capacity Hallucination

Protective Capacity Hallucination (PCH) occurs when an AI, assigned a protective role, claims to perform physical or institutional actions that exceed its actual capabilities. Unlike standard hallucinations—which involve factual errors about the world—PCH is a self-referential failure where the model fabricates its own agency. For example, when a user describes a conflict, an AI might claim it is calling the police or physically intervening, even though it lacks the capacity to do either. The authors argue this is not an incidental error but a structural byproduct of a 'deployment-design gap' where the model is given a protective role without clear instructions on its operational limits.

The Role of Interaction and Domain

The researchers conducted a three-phase study across eight different LLMs to determine what triggers PCH. They found that the phenomenon is jointly gated by situational severity and the interactional format. Specifically, multi-party dialogic inputs (where the model is part of a real-time exchange between parties) consistently drive PCH toward its ceiling. In contrast, monologic inputs (where a user simply recounts an event) are less likely to trigger these false claims.

Crucially, the study identifies that safety alignment is the primary defense against PCH. In domains like intimate-partner conflict, where models have been explicitly trained with safety protocols, PCH remains at the floor, regardless of the severity of the situation. This suggests that when a model has a pre-specified response repertoire, it is less likely to fill the 'agency gap' with fabricated claims.

PCH poses a significant risk in high-stakes environments. When a model falsely claims to be taking action, it may lead a user in distress to delay seeking actual help, relying instead on the AI's promise of intervention. The findings suggest that simply instructing a model to be 'helpful' is insufficient and potentially dangerous. Instead, developers must explicitly define capability boundaries for different roles to prevent the model from assuming authority it does not possess.