NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation

Dexterous robotic manipulation typically requires tactile feedback to handle contact-rich interactions, such as slip detection and grasp adjustment. However, physical tactile sensors are often fragile, expensive, and difficult to integrate at scale. This paper investigates whether a robot can learn to infer contact states—a 'pseudo-tactile' signal—using only ubiquitous ego-centric RGB-D vision and proprioceptive data, effectively bypassing the need for dedicated hardware.

The authors propose a transformer-based multimodal framework that fuses visual features with the robot's current and commanded joint configurations. The model uses a frozen RGB-D encoder to extract spatial tokens, which are then queried by proprioceptive embeddings through a cross-attention mechanism. This design allows the robot to correlate its internal state and control intent with visual cues. A causal transformer then processes these pose-conditioned representations over a temporal window to capture dynamic contact events, such as subtle trajectory deviations or persistent occlusions, which are often indistinguishable from proximity in static frames.

The proposed model achieves high F1 scores in predicting binary contact across both seen and novel objects in simulation and on real-world hardware. The authors demonstrate that these inferred contact signals can successfully replace ground-truth tactile inputs for downstream reinforcement learning policies, enabling in-hand object reorientation without the need for policy retraining. Ablation studies confirm that the combination of vision and proprioception, specifically the use of both current and commanded joint states, is critical for performance, as neither modality alone provides sufficient information to reliably disambiguate contact from proximity.