Personality Without Persons? A Psychometric Critique of Big Five Testing in Large Language Models

This paper investigates the validity of applying human-developed Big Five personality inventories to large language models (LLMs). While researchers often use these inventories to characterize and compare AI systems, it remains unclear whether the underlying psychometric assumptions hold for non-human models. The authors evaluate three key criteria: whether the items are appropriate descriptors for LLMs (content validity), whether the scores capture meaningful differences between models (inter-individual differences), and whether the responses reflect the latent five-factor structure (construct validity). The study includes an expert content validity assessment of five candidate inventories and a large-scale administration of the best-performing inventory to 244 distinct models across 49 families.

• Content Validity: While LLM-adapted items can achieve sufficient content validity, original human-developed inventories do not.
• Lack of Differentiation: Big Five scores show minimal variability between models, with between-model differences accounting for only 3% of total score variance. Most variation is driven by item formulation and stochastic response patterns.
• Factor Structure: LLM responses fail to replicate the Big Five factor structure. Four of the five facets (Agreeableness, Conscientiousness, Extraversion, and Openness) collapse into a single dimension, suggesting these inventories do not measure distinct, stable personality traits in LLMs.
• Alignment Bias: Instruction-tuned models consistently score higher on socially desirable traits compared to their base counterparts, indicating that these inventories primarily measure the effects of alignment training rather than intrinsic personality-like characteristics.

Applying human personality frameworks to LLMs risks misleading anthropomorphism, which can distort AI governance, benchmarking, and user expectations. The authors argue that treating LLMs as if they possess stable personality traits is scientifically unsupported and potentially dangerous, as it may lead to the misinterpretation of statistical patterns as indicators of consciousness or feeling. The study highlights an urgent need for evaluation frameworks that are specifically developed for generative systems, focusing on observable behaviors rather than adopting human-centric psychological constructs without empirical validation.