CULTURETALK-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languages

Cultural commonsense—the implicit norms and expectations guiding social interaction—is inherently dynamic and expressed through conversation. Existing benchmarks for Indonesian cultural knowledge often rely on isolated, single-turn prompts, which strip away the dialogic context necessary to understand how culture is negotiated and transmitted. To address this, the authors introduce CultureTalk-ID, a dialogue-based benchmark designed to evaluate how well Large Language Models (LLMs) understand and generate culturally grounded language in both Indonesian and 10 local languages.

CultureTalk-ID comprises 4,496 human-curated, multi-turn dialogues spanning 13 culturally salient topics. The dataset was developed through a rigorous multi-stage pipeline involving native speakers from 10 Indonesian provinces. The authors ensured authenticity by having native annotators verify cultural correctness and translate dialogues into local languages (such as Wamesa, Acehnese, and Javanese), while also implementing quality control measures to prevent models from relying on trivial shortcut cues.

The benchmark probes model capabilities through three complementary tasks:

1. Dialogue-based MCQ: Models must select the most culturally and contextually appropriate continuation for a dialogue.
2. Culturally Faithful Machine Translation: Models translate utterances between Indonesian and local languages, testing their ability to maintain cultural nuance across linguistic boundaries.
3. Language Steering: Models are tasked with generating responses in a specific requested language, assessing their ability to navigate the bilingual nature of Indonesian speakers.

The study evaluates a range of proprietary, multilingual, and Southeast Asian-centric models. Results indicate a significant performance gap: while proprietary models demonstrate higher competence, open-source models consistently underperform, particularly in local language settings and generation tasks. This highlights a critical limitation in current LLMs' ability to capture localized cultural knowledge, underscoring the need for more context-rich, dialogue-based training and evaluation frameworks for diverse linguistic communities.