LLM-based Multimodal Personality Recognition via Facial Action Unit-Text Semantic Fusion

Personality recognition in asynchronous video interviews (AVIs) is often limited by unimodal approaches that ignore the interplay between verbal and non-verbal cues. This paper investigates whether semantically fusing facial action units (AUs) with textual interview responses can provide a more robust and psychologically grounded framework for assessing personality traits under the HEXACO model.

The authors introduce a four-component framework: (1) Preprocessing, which uses a keyframe-centered windowing strategy to preserve local temporal dynamics; (2) AU selection, which employs a multi-objective simulated annealing approach to identify the most informative and compact subset of AUs; (3) Semantic description generation, where an LLM converts numerical AU sequences into interpretable natural language; and (4) Personality trait prediction, where a LoRA-fine-tuned LLM fuses these AU-derived descriptions with textual interview transcripts. A lightweight regression head is then used to map the resulting embeddings to continuous personality scores, effectively decoupling semantic understanding from numerical estimation.

Experiments on the AVI-6 benchmark demonstrate that the proposed AU-text fusion framework consistently outperforms both unimodal baselines and traditional multimodal fusion methods. The model achieves lower prediction errors and stronger correlations with human-rated personality scores across all four HEXACO traits (Honesty-Humility, Extraversion, Agreeableness, and Conscientiousness). The ablation studies confirm that both the AU-derived semantic descriptions and the textual responses provide complementary information, and that decoupling the semantic representation from the regression head is critical for training stability and predictive accuracy.

This research bridges the gap between low-level facial behavioral data and high-level linguistic reasoning in personality assessment. By grounding non-verbal cues in interpretable semantics, the framework provides a scalable and explainable method for automated personality recognition, offering a more nuanced understanding of how individuals manifest personality traits in formal interview settings.