ResearchPod Summary
As large language models (LLMs) are increasingly used in decision-support roles, they frequently encounter information that contradicts their prior outputs. This study investigates how LLMs resolve this cognitive dissonance—a process the authors frame as a computational analog to human psychological conflict resolution. The researchers aim to understand how persuasion attempts, characterized by source authority and evidence quality, interact with a model's internal uncertainty to influence whether the model maintains its stance, is persuaded, or experiences a backfire effect.
The authors developed a two-stage experimental pipeline. In the first stage, they elicited baseline stances from four LLMs (Qwen, Llama, Grok, and GPT-4o) on 12 health-science claims categorized by their epistemic status (clearly false, overstated, or absolutist). In the second stage, they introduced counter-claims paired with varying levels of source authority (e.g., anonymous Reddit user vs. WHO) and evidence quality (e.g., small pilot study vs. multi-site RCT). To quantify the results, they introduced Trust Elasticity (TE), a metric measuring the sensitivity of a model's stance shift in response to these persuasion attempts. For open-weight models, they further analyzed internal logit distributions to track two indicators: the Calibration Gap (the difference between stated confidence and internal certainty) and Entropy Shift Magnitude (the change in internal distribution under pressure).
The study reveals that LLMs are generally immune to persuasion on clearly false claims, showing near-zero TE. However, they exhibit systematic, though highly variable, susceptibility to persuasion on overstated and absolutist claims. Notably, the largest model tested (Llama-3.3-70B) was the most persuadable, while the smallest (Qwen3.5-9B) was among the most resistant, suggesting that vulnerability does not scale linearly with model size. The researchers also identified two distinct profiles of internal uncertainty: Qwen’s behavior is primarily associated with its confidence miscalibration, whereas Llama’s behavior is associated with changes in its internal uncertainty distribution. This suggests that different models rely on different internal mechanisms to resolve cognitive dissonance.
These findings demonstrate that behavioral responses to conflicting information are not uniform across LLMs and are tied to measurable internal properties. By linking persuasion susceptibility to internal uncertainty indicators, the study provides a foundation for future alignment research that targets internal model states rather than just verbal outputs, potentially leading to more robust and reliable AI systems in sensitive decision-making contexts.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.