Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI

This paper investigates how vision-language models can be extended from third-person video understanding to first-person egocentric video, with a specific focus on bridging hand-object interactions and embodied artificial intelligence. The authors conduct a structured survey tracing the evolution of models from traditional convolutional architectures to multimodal foundation models. Their approach centers on explicit relational reasoning, such as scene graphs and hand-object interaction graphs, as a primary organizing principle to address the unique perception challenges of wearable video.

First-person video introduces severe technical hurdles that standard exocentric vision models cannot resolve. Continuous camera motion and head jitter generate sudden blur and erratic frame transitions. Furthermore, acting hands and closely held objects lead to frequent self-occlusions and partial visibility, while daily routines demand the fine-grained differentiation of subtle manipulation stages over extended time horizons. The authors argue that conventional global vision-language models collapse when evaluated on these fine-grained action compositions because they lack structured temporal and relational grounding.

The survey connects egocentric perception directly with robotic skill learning and assistive systems. By treating wearable video as a scalable source of embodiment data, researchers can co-train human and robot manipulation policies. However, closing the gap between raw first-person observations and language-grounded robotic decision-making requires moving beyond appearance-based recognition toward temporally grounded, relationally explicit reasoning systems that can interpret human goals across long-form activities.