Learning to Detect Cross-Modal Negation: An Analysis of Latent Representations and an Attention-Based Solution

This paper investigates why current vision-language models (VLMs) struggle to detect negation—a fundamental semantic concept—across different modalities. The authors hypothesize that negation is not inherently captured by standard pretrained embeddings, which tend to focus on modality-specific features rather than cross-modal semantic relationships. To test this, they analyze 3,222 political video-text pairs, using linear discriminant analysis (LDA) and various classification tests to assess the separability of negation signals in latent spaces. They then propose a cross-modal attention architecture designed to explicitly model dependencies between text and video, comparing its performance against unimodal baselines and testing various attention head configurations.

The analysis reveals that negation does not form a separable class in the latent spaces of standard VLMs, with most models performing at chance levels when tasked with distinguishing negated from non-negated content. The authors identify a key asymmetry: while textual negation often functions independently, visual negation is semantically dependent on linguistic context. Their proposed cross-modal attention framework effectively bridges this gap, achieving significant performance gains (up to +7.03% F1) by forcing the model to align and fuse signals from both modalities. Furthermore, the study finds that using specialized video representations, such as those from JEPA-V2, can further enhance the model's ability to capture temporal negation cues.

Understanding negation is critical for robust multimodal reasoning, particularly in political discourse where contradictions between speech and visual evidence are common. By demonstrating that current VLM embeddings lack a generalizable negation signal, this work highlights a significant limitation in existing representation learning paradigms. The proposed cross-modal attention mechanism provides a practical, architecture-level solution that improves semantic alignment, offering a path forward for building more reliable AI systems capable of interpreting complex, contradictory, or nuanced multimodal information.