Efficient and Interpretable Body-Based Emotion Recognition with Lightweight Temporal Convolutional Networks

This paper investigates whether lightweight Temporal Convolutional Networks (TCNs) can serve as an efficient, interpretable alternative to graph-based skeleton models for body-based emotion recognition. While graph-based models are effective at capturing the spatial structure of the human body, they often introduce significant computational overhead. The authors evaluate a family of TCN architectures on the DIEM-A dataset, comparing them against a graph-based time-series graph (G-TSG) baseline. The study measures performance via accuracy and macro-F1, while assessing efficiency through parameter counts, memory footprint, and inference latency. Additionally, the authors explore model interpretability by analyzing how specific body regions contribute to classification using region-specific training, zero-based occlusion, and gradient saliency.

The results show that the TCN-Base model provides a highly favorable trade-off between performance and efficiency. While the G-TSG baseline achieved slightly higher mean performance, the TCN-Base remained within 1.58 accuracy points and 1.25 macro-F1 points of the baseline. Crucially, the TCN-Base utilized 79.18% fewer parameters and reduced inference latency by approximately 12.5 times compared to the G-TSG. Statistical tests (Wilcoxon signed-rank and paired t-tests) indicated that the performance gap between the two models was not statistically significant. Regarding interpretability, the study found that upper-body motion provides the strongest standalone cue for emotion recognition, though the importance of specific body regions varies depending on the emotion being classified.

Real-time affective systems, such as social robots and virtual agents, require low-latency emotion recognition to function effectively in interactive environments. By demonstrating that lightweight TCNs can achieve competitive performance with significantly lower computational costs, this research provides a practical path for deploying emotion-aware systems on resource-constrained hardware. Furthermore, the multi-method approach to interpretability offers researchers a clearer understanding of how motion cues are utilized by models, helping to bridge the gap between black-box predictions and human-understandable affective movement analysis.