Mohamed Adjel, Kahina Chalabi, Vincent Bonnet
8 min
Abstract
Real-time 3D human mesh recovery from a single image on mobile devices is a challenging problem with many applications, including robotics, biomechanics, AR/VR, and exergames. Current state-of-the-art models are either too slow to run efficiently on mobile devices or sacrifice accuracy for speed. We present InstantHMR, a family of open-source single-image regressors suited for mobile inference, based on the Momentum Human Rig (MHR) parametric model and trained exclusively on publicly available annotations from the SAM 3D Body dataset. Our approach achieves remarkable performance on a Samsung Galaxy S23 NPU, recovering a 6,890-vertex body mesh in 5.5 ms per person, and 0.98 ms on an NVIDIA RTX 4070 GPU. This speed surpasses all existing methods we evaluated. On mobile, it is 12x faster than the single mobile-compatible method that slightly outperforms ours in accuracy, while achieving competitive accuracy with 40.0 mm PA-MPJPE on 3DPW, 48.8 mm on EMDB and 41.7 mm on RICH. To ensure fair comparisons, we re-evaluate 11 state-of-the-art methods under a unified protocol aligned with the closest MHR-based competitors, including Meta's official MHR-to-SMPL conversion, and report their reproduction results. Additionally, we measure latency across eight different targets, ranging from desktop TensorRT to two phone NPUs. Code and models are available at https://github.com/Naturalpad-studio/InstantHMR.
Alex: So hardware fit is part of the method. What example shows why that matters beyond just having fewer parameters?
Sam: Consider NLF, an accurate competing method. It estimates surface points through separate spatial queries, then combines scores to locate each point. On the phone accelerator, much of its time goes into moving, rearranging, and reducing large tensors rather than convolutions. InstantHMR instead predicts the rig controls once, then generates the surface through fixed body-model calculations.
Alex: The same workload can therefore look quite different on a desktop and a phone. Does their direct comparison support that interpretation?
Sam: Yes. InstantHMR and the smaller NLF are fairly close on the phone's main processor. But on its neural accelerator, NLF is about twelve times slower. Both run entirely on that accelerator, so the authors can rule out processor fallback as the explanation for this gap.
Alex: Where did the training supervision come from? A compact model could inherit the limitations of whatever larger model labelled its examples.
Sam: They used released human-annotated body fits from the SAM 3D Body dataset, rather than a teacher model's predictions. Training covered about four million person crops drawn from several image sources. None of the benchmark images used for their evaluation were in InstantHMR's training data.
Alex: Was performance mainly about architecture, or did the training recipe change the result too?
Sam: The training recipe mattered substantially. Most training images used the standard person crop, but others were zoomed, shifted, or partly occluded. Mixing those framings produced the largest single improvement the authors measured in their ablations. It teaches the model under different views of the crop, rather than assuming the person is always framed the same way.
Alex: Now the accuracy comparison. These systems predict different body representations, so a leaderboard could partly measure how outputs were converted.
Sam: That's a major concern here. Person boxes, camera focal lengths, joint definitions, and body-model conversions can all move the reported error. The authors re-evaluated eleven competing methods under a mostly matched protocol. For systems using their body rig, they applied Meta's official conversion to the common comparison model.
Alex: You said mostly matched. What exception should we keep in mind?
Sam: NLF used its own person detector, following its published setup, rather than the benchmark person boxes. The authors checked that changing those boxes had only a small effect for NLF. They also report whether their reproduced scores agree with each competitor's paper, rather than treating every published number as directly interchangeable.
Alex: Give us one accuracy result we can actually interpret, rather than a string of benchmark scores.
Sam: On EMDB, one held-out benchmark, InstantHMR's medium model has an aligned average joint-position error of about forty-nine millimetres. The smaller NLF scores about forty-five millimetres under this evaluation. Lower is better, but this metric first aligns the reconstructed body to the reference, so it isn't an absolute positioning guarantee.
Alex: A modest accuracy loss there buys a much shorter wait on the tested accelerator. How broad is the evidence behind that comparison?
Sam: They evaluate across three body-reconstruction benchmarks and measure latency across desktop and phone targets. On the tested phone accelerators, InstantHMR is the most accurate method among those meeting their real-time threshold of thirty frames per second. My reading is that the model-level trade-off is well supported, especially because they rerun competitors rather than just borrowing headline scores.
Alex: But model-level timing is not camera-to-feedback timing. What does the stopwatch leave out?
Sam: Detection and cropping are excluded for every method. The timing starts with the person crop and includes producing the posed surface. It doesn't include the official body-model conversion used for benchmark scoring. So this supports a fast reconstruction component, not a measured end-to-end application.
Alex: For movement feedback, I'd also care whether the body flickers between frames. Did they test that?
Sam: No. The authors report no temporal metric for jitter or acceleration error. Each frame is estimated independently, and evaluating stability, including with their demo's smoothing filter, is future work. That's the most important gap if your application depends on smooth motion rather than isolated poses.
Alex: What else should stop a researcher from treating this as ready for any phone or product?
Sam: Accelerator measurements cover Qualcomm-equipped Android phones, not Apple's Neural Engine. Hands receive no dedicated close-up processing. And the main weights are research-only because some training images have restrictive licences. A commercially usable variant exists, but the authors report substantially worse accuracy with its smaller training corpus.
Alex: With those boundaries, who should read the full document, and where should they begin?
Sam: Researchers building mobile body estimators should start with the evaluation protocol, then the explanation of why InstantHMR is fast on the phone accelerator. Read the limitations before planning a live application. For everyone else, remember this: a model isn't fast on a phone just because it's fast on a desktop. The operations have to fit the chip.
Alex: Keep that distinction in mind when choosing your next baseline.