Soroush Mehraban, Vida Adeli, Babak Taati
7 min
Abstract
Recent transformer-based approaches have demonstrated excellent performance in 3D human pose estimation. However, they have a holistic view and by encoding global relationships between all the joints, they do not capture the local dependencies precisely. In this paper, we present a novel Attention-GCNFormer (AGFormer) block that divides the number of channels by using two parallel transformer and GCNFormer streams. Our proposed GCNFormer module exploits the local relationship between adjacent joints, outputting a new representation that is complementary to the transformer output. By fusing these two representation in an adaptive way, AGFormer exhibits the ability to better learn the underlying 3D structure. By stacking multiple AGFormer blocks, we propose MotionAGFormer in four different variants, which can be chosen based on the speed-accuracy trade-off. We evaluate our model on two popular benchmark datasets: Human3.6M and MPI-INF-3DHP. MotionAGFormer-B achieves state-of-the-art results, with P1 errors of 38.4mm and 16.2mm, respectively. Remarkably, it uses a quarter of the parameters and is three times more computationally efficient than the previous leading model on Human3.6M dataset. Code and models are available at https://github.com/TaatiTeam/MotionAGFormer.
Sam: At the heart is the AGFormer block, built around MetaFormers—a flexible setup where Transformers get upgraded by swapping their core attention mixer for any info-blending module. Each stream has a spatial stage, treating individual body joints as separate tokens to capture relationships within a single frame, then a temporal stage reshaping to treat full frames as tokens for links across time. The transformer stream mixes using multi-head self-attention, weighing distant connections heavily. The GCNFormer stream specializes in local ties using graph convolutions, where joints are nodes and connections are edges defined by adjacency matrices. In spatial, it uses a fixed human skeleton topology—like predefined bone links—to enforce anatomical closeness. For temporal, it builds a data-driven graph by finding the K nearest neighbors between a joint's features across frames, based on their similarity scores, letting the model learn motion-specific neighborhoods.
Alex: That explains why GCNs don't blur local structure—they're wired directly to joint realities. With parallel streams running these, how do they avoid just duplicating effort?
Sam: Running spatial and temporal MetaFormers in parallel per stream preserves unique strengths: transformers grab broad patterns across all joints or frames, while GCNs sharpen nearby anatomical and motion constraints. The paper highlights this lets the model better integrate local details that transformers might smooth over, without needing strided reductions that limit global scope in prior GCNs. Evidence from ablations backs that both contribute distinctly.
Alex: If inputs rely on 2D detectors, noisy video from a shaky phone could mess up those graphs?
Sam: Yes, the paper implies performance ties to solid 2D inputs, and erratic motions might challenge the temporal K-NN graph. They test on Human3.6M, a huge indoor dataset with 3.6 million frames from 11 people doing daily activities—training on five subjects, testing on two others for fairness. The main metric is MPJPE, the average distance in millimeters between predicted 3D joint spots and the real ones, after aligning the body roots. On estimated 2D inputs, MotionAGFormer-B gets 38.4 mm P1 error, beating MotionBERT's 39.2 despite a quarter of the parameters and three times lower computation per frame.
Alex: So lower millimeters mean better 3D guesses, like nailing where elbows and knees really are in space. How does it do on the other dataset?
Sam: MPI-INF-3DHP mixes green screen, regular rooms, and outdoors for real-world variety, scored on MPJPE plus correct keypoints within 150 mm and area under the curve. They adjust to 81 frames due to shorter videos and outperform others across MPJPE.
Alex: This dual-stream setup not only shrinks the model but squeezes better accuracy from less data. Those margins highlight the lightweight edge without sacrificing much.
Sam: Correct—the paper ties gains to accurate 2D poses, and erratic motions could strain temporal graphs. The paper includes visualizations comparing MotionAGFormer-B to recent methods like STCFormer, PoseFormerV2, and MotionBERT on Human3.6M samples, showing tighter alignment with ground truth poses. They also visualize the temporal GCNFormer's adjacency matrices for joints like the left ankle or knee, revealing how early layers capture sequence-specific links that evolve deeper in, favoring connections to nearby frames.
Alex: So the graphs learn to prioritize a joint's motion in adjacent moments, like tracing a knee's path smoothly? What do the ablation studies reveal about why this hybrid shines?
Sam: Ablations on Human3.6M confirm deeper but narrower models perform best—a 16-block setup at 128 channels matches 12 blocks at 256 but uses about three times less memory and computation. Swapping GCNFormer for plain GCNs or TCNs worsens P1 error by 0.7 to 0.9 mm. Parallel fusion outperforms sequential by preserving distinct contributions, dropping P1 from 43.6 mm transformer-only to 38.4 mm hybrid.
Alex: That underscores the parallel design's smarts—each stream stays in its lane without crossover blur. Overall, MotionAGFormer advances 3D pose estimation by fusing transformer's global patterns with GCNFormer's local precision in an efficient parallel block, enabling scalable variants that outperform priors on key benchmarks.
Alex: It positions phone-based motion capture as practical for AR fitness or vehicle safety, balancing speed and detail thoughtfully. That's our look at MotionAGFormer and efficient 3D pose estimation. Thanks for tuning into ResearchPod.