Beyond Pairwise Relations: Dynamic Manipulation Hypergraphs for Vision-Based Human Activity Recognition

Fine-grained manipulation activities—such as cutting, fastening, or opening—are defined by the coordinated actions of hands, objects, tools, and surfaces. Traditional graph-based models often represent these interactions through pairwise edges, which can fragment a single, complex event into disconnected binary relations. This paper asks whether explicitly modeling these multi-entity configurations as higher-order relational units can improve the accuracy of human activity recognition (HAR) in egocentric video.

The author introduces a dynamic manipulation hypergraph framework that represents video clips as sequences of hypergraphs. At each temporal step, the model identifies manipulation-relevant entities (hands, objects, tools, surfaces) and encodes them using appearance, spatial, motion, and semantic-role features.

Instead of simple edges, the model instantiates hyperedge candidates based on proximity, contact, and motion-coupling predicates. A hypergraph reasoning network then performs message passing between nodes and hyperedges, followed by temporal attention to capture the evolution of the interaction. The model also generates class-agnostic hyperedge-importance scores, which provide insight into which entity configurations and time intervals the model prioritizes during classification.

The framework was evaluated on the EPIC-KITCHENS-100/VISOR and Assembly101 datasets. The dynamic hypergraph approach significantly outperformed both a matched pairwise graph baseline (improving HO-F1 by 6.9 and 9.5 percentage points, respectively) and a static hypergraph baseline. Qualitative analysis on the ARCTIC dataset further confirmed that the model’s highest-ranked hyperedges align well with contact-rich intervals in the video, demonstrating that higher-order relational modeling effectively captures the structure of complex manipulation tasks.