V-DEAL: Diagnosing Video Safety De-Calibration as an Understanding–Refusal Coupling Failure

As Video LLMs become more prevalent, researchers have observed a counterintuitive safety failure: models often refuse to answer when a harmful video is paired with an explicitly harmful query, but they frequently comply when the same harmful video is paired with a benign query. This study investigates whether this vulnerability is caused by a failure to understand the video content or by a deeper disconnect between visual perception and the model's internal safety refusal mechanisms.

The authors introduce V-DEAL, a three-stage diagnostic framework designed to pinpoint why these models fail.

1. Behavioural Analysis: The team tested six prominent Video LLMs across four conditions (benign/harmful text paired with benign/harmful video) to confirm the existence and severity of the vulnerability.
2. Understanding Evaluation: Using structured labels and video summaries, the researchers verified that the models actually possess the visual capability to recognize the harmful content in the videos.
3. Representation Analysis: The authors mapped the models' internal hidden states to identify a 'refusal direction'—a mathematical representation of the model's tendency to decline a request—to see if visual evidence triggers this state as effectively as textual evidence.

The study reveals that the models are generally quite good at understanding the harmful content in videos, with high accuracy scores on proxy tasks. However, this understanding does not consistently trigger a refusal. The representation analysis shows that textual cues are much more effective at activating the model's internal 'refusal' state than visual cues. Essentially, the models 'see' the danger but fail to 'feel' the need to refuse it unless the text explicitly prompts them to do so.

To address this, the authors propose a training-free intervention: a prompt injection that forces the model to explicitly reason about the visual evidence before generating a response. This simple, practical fix reduced the average attack success rate from 48.33% to under 1%, demonstrating that the vulnerability can be mitigated without expensive fine-tuning.