Learning to Deny: Action Denial in Multimodal Large Language Models

Modern Multimodal Large Language Models (MLLMs) excel at recognizing actions when they occur, but their ability to correctly deny an action when it is absent—despite the presence of typical contextual cues—remains largely untested. This paper investigates whether MLLMs truly understand the causal relationship between motion and action, or if they rely on spurious correlations between objects, scenes, and action labels.

The authors introduce UCF101-AD, a large-scale benchmark containing 11,283 video clips paired into Action-Presence and Action-Denial sets. In the Action-Denial clips, the context (actors, objects, and locations) is identical to the positive counterpart, but the defining motion is explicitly absent. The authors evaluate 20 state-of-the-art MLLMs on this dataset and propose CausalAct, a causal framework that represents scene structure as a directed acyclic graph (DAG). This framework uses natural language prompts to guide models to verify the causal chain—from context to interaction to motion—before predicting an action. They further improve performance by finetuning smaller models on auxiliary graph-reasoning tasks that do not reveal the target action labels.

Evaluation reveals a consistent failure across all tested MLLMs: while models often exceed 85% accuracy on positive action recognition, their performance on action denial frequently drops below 50%. This suggests that models are prone to sycophancy and shortcut learning, where they affirm plausible actions based on context rather than verifying the actual motion. The authors demonstrate that this failure is linked to a positive correlation between shortcut learning and sycophantic behavior. By incorporating CausalAct, models are better able to reject false positives, shifting their behavior from context-based guessing to verifying the causal evidence required for an action to occur.