Temporal Preservation over Processing: Diagnosing and Designing Spatiotemporal Single-Stage Video Detectors

This paper investigates whether single-stage video detectors genuinely utilize temporal context or merely exploit single, highly informative frames. To address this, the authors introduce two main contributions: YOLO-3D, a modular spatiotemporal architecture built on YOLOv8, and TemporalLens, a diagnostic framework that uses controlled perturbations (such as occlusions, shuffling, and resolution degradation) to probe temporal reliance. By comparing stacked-frame 2D models with 3D spatiotemporal models, the authors isolate the architectural factors that enable meaningful sequence integration.

The authors identify that aggressive temporal downsampling in conventional 3D backbones is a major bottleneck, often collapsing the temporal dimension to a single frame before features reach the neck. Their proposed temporal-preserving backbone, which maintains uniform temporal depth across pyramid levels, yields the largest performance gains (+3.7 pp mAP@50 at 32 frames). The diagnostic framework reveals that while stacked-2D models often collapse when the target frame is hidden, the 3D architecture successfully recovers predictions from earlier frames, confirming genuine temporal reasoning. Interestingly, the study finds that once temporal information is preserved in the backbone, simpler downstream modules (like concatenation-based fusion) are often sufficient to achieve peak performance.

Standard accuracy metrics like mAP fail to distinguish between models that use temporal context and those that rely on single-frame shortcuts. This research provides a principled way to measure temporal reasoning, allowing researchers to make informed decisions about whether the computational cost of 3D convolutions is justified for their specific task. By reframing the design of video detectors as a trade-off between temporal robustness and compute, the authors provide a roadmap for building efficient, real-time video analysis systems.