Pancasila-Dilemmas: Evaluating Large Language Models on Indonesian Human Value Dilemmas Grounded in Pancasila

As large language models (LLMs) are increasingly deployed globally, there is a growing need to ensure they align with local cultural norms rather than just Western-centric values. This paper investigates how well 50 diverse LLMs navigate moral dilemmas specific to the Indonesian context, using the national ideology of Pancasila—comprising Religion, Humanity, Unity, Democracy, and Social Justice—as the evaluation framework.

The authors introduced the Pancasila-Dilemmas dataset, consisting of 1,834 multiple-choice questions derived from real-world Indonesian news articles. Each scenario presents a moral conflict, and the dataset was annotated by 185 diverse Indonesian citizens to capture a spectrum of human perspectives. The researchers evaluated a wide range of models, including closed-source APIs and open-source models, using two primary metrics: the Probabilistic Match Score (PMS), which measures alignment with the distribution of human preferences, and the Max-Vote Agreement Score (MVAS), which measures alignment with the majority consensus.

The results reveal a significant gap in cultural alignment. No model achieved high scores, with most models failing to exceed a 0.50 PMS or a 0.72 MVAS. The models struggled most notably with dilemmas involving Religion and Unity. While larger models generally outperformed smaller ones, even state-of-the-art models showed substantial misalignment with Indonesian public preferences. Interestingly, prompting models to adopt a specific Pancasila-aligned persona did not consistently improve performance compared to universal prompts, suggesting that the models lack the underlying cultural reasoning required to navigate these specific societal nuances.

This research highlights the limitations of current value-alignment techniques when applied to non-Western, nation-specific contexts. By providing a benchmark grounded in local news and societal dilemmas, the authors offer a critical tool for developers aiming to deploy AI responsibly in Indonesia, emphasizing that "universal" alignment is insufficient for capturing the complex moral fabric of specific nations.