PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought

Traditional Multimodal Large Language Models (MLLMs) often struggle with "cognitive fragility" when interpreting pointing gestures, as they tend to rely on holistic linguistic associations rather than explicit spatial geometry. This paper investigates how to improve the precision of pointing-based visual grounding by enabling models to "think with images" through a structured, iterative reasoning process.

The authors propose PointVG-R, a reasoning-guided MLLM that decomposes the pointing localization task into a four-step Visual Chain-of-Thought (V-CoT): (1) hand detection, (2) fingertip keypoint extraction, (3) geometric ray construction, and (4) target bounding box alignment.

To train this model, the authors developed the EgoPoint-CoT dataset, which contains detailed reasoning trajectories. The training process follows a two-stage paradigm:

1. Cold-start SFT: Supervised Fine-Tuning (SFT) is used to inject foundational geometric reasoning priors into the model.
2. Adaptive RL: The model is further aligned using an improved Group Relative Policy Optimization (GRPO) algorithm. To address the challenge of noisy learning signals in reinforcement learning, the authors introduced an Adaptive Importance Weighting strategy. This strategy dynamically adjusts the influence of reward signals based on the variance within sampled groups, ensuring that the model prioritizes more informative trajectories while maintaining training stability.

PointVG-R demonstrates significant improvements over existing MLLM baselines, achieving a 15.86-point increase in mean Intersection over Union (mIoU) on egocentric pointing benchmarks. The results suggest that by internalizing explicit geometric reasoning—such as projecting a ray from the fingertip to the target—the model overcomes the limitations of black-box mapping and achieves more robust, interpretable localization.

AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.