Beyond Gestures: Estimating Full Hand Pose and Contact Forces from Wrist-Worn Pressure Sensor Array | Svetoslav Kolev et al. | ResearchPod

Transcript: Beyond Gestures: Estimating Full Hand Pose and Contact Forces from Wrist-Worn Pressure Sensor Array

Sam: A wristband can estimate how your fingers move and how hard they press. In this study, pressure measured around the wrist supports both predictions, without sensors on the fingers.

Alex: Those are different things to recover. Is the claim that one wearable can follow hand movement and contact forces continuously?

Sam: Yes, with models trained separately for each wearer. This is a methods preprint from Svetoslav Kolev and colleagues at Meta Reality Labs Research, titled “Beyond Gestures.” Their wristband estimates hand shape and distributed contact force, rather than just recognizing a gesture.

Alex: That matters for virtual interaction and robot learning, where knowing someone grasped an object isn't the same as knowing their grip.

Sam: Those are applications the authors motivate. Their proposed role is alongside cameras, adding force information and covering gaps when hands disappear from view. We'll unpack the pressure signal, the evidence behind both predictions, and why the strongest result still needs a practical follow-up.

Alex: Start with what was missing. Wrist-worn hand sensing already has several approaches, so what does this add?

Sam: Earlier wrist-pressure systems mostly classified discrete gestures or estimated aggregate grip force. Some recovered partial finger movement. Continuous, fully articulated hand pose had been demonstrated using pressure, but with a full-forearm sleeve rather than a wristband.

Alex: So the novelty isn't simply sensing a squeeze. It's recovering individual finger configurations and where contact force is distributed?

Sam: That's the authors' claim. They predict force for individual fingers and individual tactile sensing regions. They also test whether independently measured hand pose improves those force estimates.

Alex: What makes pressure at the wrist informative about fingers touching a bottle somewhere else?

Sam: The method is force myography: sensing tissue deformation caused by muscle and tendon action. Forearm muscles contract, tendons move, and surrounding tissue changes shape and stiffness. A snug wristband picks up the resulting pressure patterns.

Alex: That sounds indirect. The band isn't measuring contact at the bottle; it's measuring mechanical consequences further up the arm.

Sam: That distinction matters. Tendons largely slide along the forearm, which produces relatively little direct change around its circumference. The useful signal comes from accompanying effects, including muscle bulging, tissue stiffening, and tendons pressing sideways against surrounding structures.

Beyond Gestures: Estimating Full Hand Pose and Contact Forces from Wrist-Worn Pressure Sensor Array

Svetoslav Kolev, Lingni Ma, Michael Goesele, Renzo De Nardi, Jakob Engel, Richard Newcombe

Sep 15, 2026·8:21·arxiv·Original Paper

Wrist-worn pressure sensingFull hand pose estimationInteraction force estimationRecurrent neural networkMultimodal sensor fusionForce Myography (FMG)

Listen

8 min

Summary

Research Overview

This paper introduces a novel wristband system designed to capture the complex hand state—both 3D hand pose and distributed interaction forces—using only pressure sensors. While vision-based systems are effective for pose, they struggle with occlusions and are blind to internal contact forces. Instrumented gloves provide high-fidelity data but are often too fragile or obtrusive for everyday use. The researchers hypothesized that the distal forearm acts as a rich sensing site where muscle and tendon activity creates specific pressure patterns, allowing for accurate inference of hand state.

Approach

The system utilizes flexible capacitive pressure sensor arrays that require no skin-electrode coupling, making them robust to sweat and consistent with everyday wear. A recurrent neural network (GRU) architecture processes the pressure data—and its temporal derivatives—to predict 20 finger joint angles, 3 wrist angles, and distributed interaction forces across the hand. The team trained the model using synchronized datasets containing motion-capture pose data and tactile-glove force measurements across various natural hand-object manipulation tasks.

Key Findings

The researchers demonstrated that pressure patterns at the wrist strongly correlate with hand pose and interaction force. Their system achieves a 4.6-degree mean absolute error (MAE) for finger-joint angles on single-user motion tasks. For force estimation during natural object manipulation, the model explains a significant portion of variance (R²=0.57 for per-finger forces). Notably, they found that feeding an external pose signal—simulating a companion egocentric camera—into the wristband model reduces force estimation error by approximately 25%, indicating that pose-conditioning helps the network disambiguate force-related pressure changes from those caused by finger articulation.

Why It Matters

This work provides a practical path toward capturing fine-grained hand and force data without the line-of-sight constraints of cameras or the physical obtrusiveness of sensorized gloves. By acting as a complementary node in a constellation of wearables, this wristband can provide force information that vision cannot see, and maintain tracking accuracy when the hand is occluded or out of frame, opening new possibilities for VR interaction and high-fidelity robot learning demonstrations.

Key Terms and Definitions

Force Myography (FMG) — A sensing modality that measures surface tissue deformation on the skin caused by underlying muscle and tendon activity.
Capacitive Pressure Array — A flexible sensor grid that detects mechanical pressure by measuring changes in electrical capacitance as layers compress.
Taxel — A single tactile sensing element within an array, used here to provide spatially resolved force data across the palm and fingers.
Pose-Conditioning — The technique of providing an externally derived kinematic state (such as 3D hand pose from vision) as an input to improve the accuracy of a second, harder-to-measure target like interaction force.
Distal Forearm — The region just proximal to the wrist joint, which the authors identify as the optimal, information-rich location for pressure-based hand-state sensing.

Abstract

Capturing hand motion and interaction forces is critical for interactive computing, VR, and high-fidelity tactile demonstrations for robot learning. We introduce a wrist-worn pressure-sensing wristband that recovers continuous full-hand pose and distributed contact force on a single wearable. The system consists of flexible capacitive sensor arrays around the wrist, which require no electrical skin contact, and a recurrent network that maps the resulting pressure signal to hand state. Our key insight is that muscle contraction and tendon displacement produce pressure patterns, which correlate strongly with hand pose and interaction force. To validate this, we collect synchronized recordings of wrist pressure, optical motion-capture hand pose, and tactile-glove interaction force, covering isolated finger motion, fingertip-force stress tests, and natural hand-object manipulation. On isolated single-user motion the wristband attains $4.6^\circ$ mean finger-joint MAE, and across four users manipulating everyday objects it estimates per-finger contact force at $R^2=0.57$, which an external pose signal brings up to $0.75$. We see the wristband as one node in a constellation of everyday wearables -- e.g. paired with an egocentric camera -- adding the contact force that vision cannot observe and taking over when the hand is occluded.

Beyond Gestures: Estimating Full Hand Pose and Contact Forces from Wrist-Worn Pressure Sensor Array: Research Paper Podcast