Context-structured Video Anomaly Detection with Large Vision-Language Models

Traditional video anomaly detection (VAD) methods typically require extensive training on annotated datasets, which is costly and often fails to generalize to new environments. While large vision-language models (LVLMs) offer a training-free alternative, they often rely on holistic inference—processing the entire video at once—which can obscure context-specific details necessary for identifying subtle anomalies. This paper asks whether decomposing video analysis into distinct contextual branches can improve anomaly detection accuracy without requiring task-specific training.

CSI-VAD (Context-Structured Inference for Video Anomaly Detection) breaks down the anomaly detection task into three independent branches, each designed to capture a specific type of information:

• Environment Module: Analyzes spatial scene conditions and typical activities using a small set of representative keyframes.
• Object-Centric Tracking Module: Uses object detection and tracking to create overlays that highlight object interactions and motion, allowing the model to focus on fine-grained behavior.
• Event-Aware Time Module: Generates frame-level captions and segments them into semantically coherent units to summarize the progression of events over time.

Each branch produces an independent anomaly score and label. The framework then aggregates these outputs using various strategies (such as Mean Ensemble or Noisy-OR for scores, and voting rules for binary labels) to produce a final, more robust video-level decision.

Experiments on the UCF-Crime and UBnormal datasets demonstrate that CSI-VAD consistently outperforms holistic baseline models. By focusing on specific contextual cues, the framework effectively suppresses false positives caused by non-anomalous visual changes (like lighting shifts) and captures anomalies that are otherwise missed in global inference. The approach remains effective across different sampling budgets, showing that structured decomposition provides a reliable, training-free alternative to traditional, data-heavy VAD methods.