Validating LLMs in social science: Epistemic threats and emerging norms

Social scientists are increasingly using large language models (LLMs) to perform labor-intensive tasks such as coding qualitative data, simulating survey responses, and measuring complex social constructs like ideology or sentiment. While these models offer significant advantages in speed and scalability compared to traditional human coding or bespoke machine learning, they introduce unique methodological risks. These include sensitivity to prompt phrasing, inherent model biases, and the potential for hallucinated outputs. Because these models act as measurement instruments, their output quality is fundamental to the validity of the research conclusions drawn from them.

This study analyzed 2,143 papers from eight flagship social science journals to identify how researchers are currently using and validating LLMs. The authors identified 50 distinct measurement tasks across 27 papers. The findings indicate that LLM-generated measurements are frequently central to the primary empirical claims of these studies, yet the rigor applied to validating these measurements is often lacking.

Validation efforts are heavily skewed toward convergent validity—comparing LLM outputs to human-labeled gold standards or other computational tools. However, even this practice is often inconsistent. Researchers frequently fail to report intercoder reliability for human benchmarks, use small or unrepresentative validation samples, or rely on other computational tools (like dictionary-based methods) that have not themselves been validated for the specific task at hand. Other essential aspects of construct validity, such as face validity or predictive validity, are rarely addressed.

Using an LLM as a measurement instrument involves numerous researcher-led decisions, including prompt design, model selection, and procedures for extracting quantitative data from free-text responses. The authors found that these choices are often made with little justification or empirical testing. Without standardized norms for conceptualization and operationalization, researchers risk losing control over what is actually being measured. The paper argues that the field must move toward more comprehensive validation frameworks that go beyond simple convergent checks to ensure that LLM-based measurements are robust and scientifically sound.