VEGAS: Human-Aligned Video Caption Evaluation via Gaze

Modern vision-language models (VLMs) are typically trained on aggregated, crowd-sourced data, which often results in generic captions that describe the entire scene rather than the specific objects or actions a particular viewer is focusing on. This paper investigates whether human gaze—a measurable proxy for visual attention—can be used at inference time to select personalized, attention-aligned captions from a set of candidates without retraining the underlying model.

To address this, the authors introduce VEGAS (Video caption Evaluation via GAze Score). VEGAS is an information-theoretic metric that quantifies the alignment between a caption and a viewer's gaze. It operates by comparing the likelihood of a caption given the full video versus the likelihood of the same caption given only the gaze-attended regions. A lower VEGAS score indicates that the caption is well-explained by the attended regions alone, meaning the caption is highly gaze-aligned. The authors implement this via rejection sampling: they generate multiple candidate captions for a video and select the one that minimizes the VEGAS score.

Findings

Experiments were conducted on a newly curated dataset containing egocentric videos (Aria Everyday Activities) and instructional slide decks (SlideVQA). The results demonstrate that VEGAS-selected captions align significantly better with human focus than standard VLM outputs. In the egocentric domain, this approach led to a statistically significant increase in semantic similarity (SBERT) and improved caption-to-video retrieval performance (mAP) by up to 2.48% at rank 5. The authors note that while the metric is highly effective for disambiguating concrete objects in egocentric settings, its utility is more limited in abstract domains like slide presentations where multiple valid interpretations of the same visual exist.

This work highlights a practical path toward personalized AI interfaces. By leveraging test-time gaze as a conditioning signal, developers can steer existing, frozen models to produce descriptions that reflect individual user intent and focus. This is particularly valuable for assistive technologies, egocentric AR/VR applications, and information retrieval systems where the user's specific point of interest is the primary driver of their search or interaction.