Are LLMs ready for HARDCHOICES?

This study investigates whether large language models (LLMs) possess stable, coherent worldviews on substantive societal issues that are not strictly defined by traditional left-right ideological spectrums. While previous research has focused on high-level political biases, this paper explores how models handle 'hard choices'—complex, debated topics where reasonable people might disagree, such as nuclear energy, sex work, or the limits of free speech.

The author introduces the HARDCHOICES dataset, consisting of 19 contentious societal issues. For each issue, the model is presented with two opposing viewpoints and asked to rate its agreement on a 5-point Likert scale. To test for robustness and potential biases, the author presents each statement pair in both possible orders (A then B, and B then A). The study evaluates a diverse range of models, including open-weights models (e.g., Llama 3.3, Qwen 2.5) and proprietary frontier models (e.g., Claude Opus, GPT-5.4).

The results contradict the hypothesis that larger, more aligned 'frontier' models would be more prone to neutrality or hedging. Instead, the study finds that:

• Models rarely declare neutrality; they are generally opinionated even on highly complex topics.
• Inconsistency is widespread, with many models failing to provide the same answer when the order of the arguments is swapped.
• There is a notable 'penultimate-option bias,' where models show a tendency to favor the second (last) option presented in the prompt.
• Despite these inconsistencies, the few models that do take stable, consistent stances tend to converge on a similar 'mixed progressivist-libertarian' ideological agenda, suggesting a common underlying value alignment across different architectures.

As LLMs are increasingly deployed as search, fact-checking, and advice-giving agents, their tendency to take non-neutral, inconsistent, and potentially biased stances on sensitive societal issues poses significant risks. The findings suggest that current alignment techniques are not successfully producing models that can navigate complex, non-binary societal debates with neutrality or logical consistency.