VistaRef: Boosting Visual Spatial Orientation Awareness for Pointing-to-Object Detection

Traditional visual grounding models often struggle with pointing-to-object detection tasks because their global attention mechanisms prioritize semantic associations over fine-grained spatial relationships. This leads to localization drift and ambiguity, especially when targets are distant or densely packed. The authors investigate how to bridge this gap by explicitly incorporating physical geometric priors into the Transformer architecture.

VistaRef introduces a hierarchical framework that models the pointing process as a physical trajectory. It consists of four main components:

• Local Hand Entity Modeling (LHEM): Uses hand-pose embeddings to isolate and purify gestural features from a dynamically proposed region around the hand.
• Geometric Ray Modeling (GRM): Transforms implicit pointing information into explicit spatial geometric features, creating a 'ray' that guides the model's attention toward the target.
• Ray-Aware Target Grounding: Uses a cross-attention mechanism to fuse the hand's geometric orientation and visual appearance with global image features.
• Orientation-Consistent Alignment Loss (OCAL): A novel loss function that enforces physical consistency between the detected hand pose and the predicted target location.

VistaRef significantly outperforms existing state-of-the-art methods in pointing-to-object detection. By transforming abstract linguistic queries into deterministic geometric rays, the model effectively models the correlation between the hand and the target. Quantitative results show a 14-point absolute gain in grounding accuracy compared to baseline models, demonstrating superior robustness in complex, cluttered environments.

This research provides a scalable way to improve human-robot and human-computer interaction. By enabling machines to understand deictic gestures (pointing) with high spatial precision, VistaRef facilitates more natural and reliable collaboration in augmented reality and robotic systems where spatial awareness is critical.