Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration

Research on large language model (LLM) bias has historically been dominated by English-centric datasets, which fail to capture the cultural nuances and regional variations of stereotypes in other languages. To address this, the authors propose a scalable, human-LLM collaborative framework designed to generate and validate culturally specific stereotype datasets.

Instead of relying solely on expensive manual labor, the researchers use LLMs to generate candidate stereotypes through targeted prompt-based injection attacks. These candidates are then validated by in-culture annotators—individuals born and raised in the target country—who rate the prevalence of each stereotype. This approach allows for the creation of high-quality, region-specific datasets with significantly reduced costs and time compared to traditional manual curation.

The authors applied this framework to construct EspanStereo, the first native Spanish-language dataset for stereotype analysis, covering Spain, Mexico, Argentina, Colombia, and Nicaragua. The dataset includes stereotypes related to race, religion, gender, sexual orientation, and age.

Analysis of the dataset reveals that stereotypes are highly localized. For example, the overlap of stereotypes between any two countries in the dataset never exceeds 21%. The researchers highlight specific examples, such as stereotypes regarding land conflicts with indigenous groups (e.g., the Wayuu in Colombia vs. the Mapuches in Argentina), which are deeply rooted in each country's unique historical and political context.

This work demonstrates that LLMs can be effectively leveraged to identify and codify culturally specific biases that are often overlooked by translation-based or English-centric benchmarks. By providing a scalable, language-agnostic framework, the authors offer a path toward more representative, cross-cultural evaluations of LLM behavior. This is a critical step for researchers aiming to mitigate social biases in multilingual models, as it highlights that 'Spanish-language' bias is not a monolith but a collection of distinct, region-specific phenomena.