When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs

This paper investigates whether large language models (LLMs) can reliably answer everyday, culturally grounded, and long-tail knowledge questions, such as those found in trivia games or quiz shows. While existing benchmarks like MMLU focus on academic or reasoning-based tasks, the authors argue these do not reflect the gaps in a model's grasp of popular culture and everyday facts. To test this, the authors introduce TriviaRoomQA, a new benchmark containing 3,300 parallel multiple-choice questions across six European languages and an additional 5,340 French-only questions, covering 288 distinct topics.

The researchers evaluated 30 open-weight LLMs ranging from 7B to 70B parameters. The evaluation was conducted using log-likelihood scoring on multiple-choice options, allowing for a precise comparison of how models perform across different languages, difficulty levels, and thematic categories.

The study reveals a clear hierarchy in model knowledge: LLMs are highly proficient at encyclopedic and scientific topics but show a marked decline in performance when faced with popular-culture questions. Even the largest models struggle with details regarding celebrities, music, and contemporary news, suggesting that this type of information is less stable or less frequently represented in training data than canonical facts.

Additionally, the authors found that models often fail to maintain consistent performance when the same question is presented in different languages. This suggests that factual knowledge is not stored in a language-agnostic way within these models. Finally, a comparison with human participants showed that while humans exhibit a gradual decline in performance as questions become more difficult, LLMs tend to show a sharp, uniform drop-off once a topic falls outside their knowledge boundary.

This research highlights a critical limitation in current LLM development: the tendency to prioritize academic and encyclopedic knowledge at the expense of the everyday cultural context that defines human interaction. By demonstrating that scale alone does not bridge the gap between encyclopedic recall and popular-culture literacy, the authors provide a new framework for evaluating how models might better serve users in diverse, real-world, and multilingual settings.