Aaditya Khanal, Junxiu Zhou
5 min
Abstract
The practical deployment gap -- transitioning from controlled multi-view 3D skeleton capture to unconstrained monocular 2D pose estimation -- introduces a compound domain shift whose safety implications remain critically underexplored. We present a systematic study of this severe domain shift using a novel Gym2D dataset (style/viewpoint shift) and the UCF101 dataset (semantic shift). Our Skeleton Transformer achieves 63.2% cross-subject accuracy on NTU-120 but drops to 1.6% under zero-shot transfer to the Gym domain and 1.16% on UCF101. Critically, we demonstrate that high Out-Of-Distribution (OOD) detection AUROC does not guarantee safe selective classification. Standard uncertainty methods fail to detect this performance drop: the model remains confidently incorrect with 99.6% risk even at 50% coverage across both OOD datasets. While energy-based scoring (AUROC >= 0.91) and Mahalanobis distance provide reliable distributional detection signals, such high AUROC scores coexist with poor risk-coverage behavior when making decisions. A lightweight finetuned gating mechanism restores calibration and enables graceful abstention, substantially reducing the rate of confident wrong predictions. Our work challenges standard deployment assumptions, providing a principled safety analysis of both semantic and geometric skeleton recognition deployment.
Sam: From a side view in 2D, both bend hips and knees. Squats keep shins more vertical; deadlifts hinge more at the hips. Without depth cues, they blend. Visualizations show gym samples collapsing inside training clusters despite being wrong, so confidence hits 99%+ on the mistake. Half the confident predictions could lead to injury.
Alex: Even perfect out-of-distribution spotting doesn't help if wrong gym inputs hide in familiar spots. Common tricks like dropout or ensembles don't fix it?
Sam: Methods like MC Dropout add randomness by turning off neurons multiple times to check doubt. Deep Ensembles use several models for agreement. They score well on AUROC, around 0.81 to 0.98. But risk-coverage curves stay flat at nearly 100% risk. The model can't separate confident errors from rare correct gym guesses.
Alex: Great at knowing gym data is off, terrible at knowing which gym predictions are off. What's the practical fallout?
Sam: Fitness apps might approve half the inputs but with 98% risk of bad feedback on loaded lifts. Wrong advice raises injury odds—abstaining is safe, but errors aren't. Wrong-Spoke Rate measures confident wrongs at around 49% zero-shot.
Alex: How do they propose fixing it?
Sam: Energy scoring works better as an out-of-distribution signal. It flags low-energy outputs as confident in-distribution, hitting AUROC 0.91 and hinting at risk reduction. Their fix is finetuned gating—like a dimmer switch on feature channels. Learnable weights multiply backbone outputs after sigmoid, finetuned on target data with under 1% extra parameters.
Alex: It boosts useful gym signals and dials down overconfident studio noise?
Sam: Yes—like tweaking volume knobs on neural pathways to amplify 2D cues and suppress misleading 3D assumptions. Finetuning lifts accuracy to 37% single-seed, 27% multi-seed on Gym2D. It cuts risk at 50% coverage from 98% to 48-56% and Wrong-Spoke Rate to 28-33%. Risk curves finally slope down for safe thresholds.
Alex: Does this hold across models?
Sam: It improves baselines too, though less on graph convolutions than attention-based transformers. Gains are strongest on distinct moves over squat-like ones. UCF101 sees Wrong-Spoke drops, but semantic gaps limit recovery. It needs target data labels—no zero-shot. Limitations include no ground-truth 3D and multi-seed averages at 27% accuracy.
Alex: This pushes toward reliable AI coaching that skips risky calls and handles most reps safely. A solid step in seeing why confidence alone doesn't mean safety. Thanks for listening to ResearchPod.