Emotion Experience, Expression, and Perception: Emotion Analysis on Multimodal Social Media Posts

This paper investigates the complex relationship between how authors experience emotions during real-world events, how they express these emotions in multimodal social media posts (text and images), and how readers perceive these expressions. While previous research has largely focused on text-only emotion analysis, this study introduces the Multimodal Multi-Emotion-Model (Mult2EMo) dataset, which includes 7,200 posts annotated by authors and readers to capture the full chain of emotion communication.

The authors collected data in two phases: an Author Phase, where participants provided posts and detailed accounts of the triggering events and their cognitive appraisals, and a Reader Phase, where third-party annotators attempted to reconstruct the author's original experience. The researchers evaluated both human readers and fine-tuned CLIP-based models on their ability to classify emotions and predict appraisal dimensions, testing performance across different modalities (text, image, and combined).

The study reveals that emotion reconstruction is a challenging task for both humans and machines. A critical bottleneck is the interpretation of the triggering event; when readers or models fail to understand the context behind a post, their ability to reconstruct the intended emotion drops significantly. Furthermore, the researchers found that while images contain emotion-relevant information, both humans and models rely heavily on text. Current multimodal fusion strategies often fail to effectively combine these signals, leading to performance that barely exceeds text-only baselines.

This work highlights that emotion analysis is not merely a classification task but a process of reconstructing private states that are often underspecified in social media content. By demonstrating that event understanding is central to this process, the paper provides a new framework for evaluating how well AI models can perform 'theory of mind' tasks, suggesting that future progress requires models that can better reason about the causal events behind expressed emotions rather than relying on surface-level textual cues.