Ben Wigler, Maria Tsfasman, Tiffany Matej Hrkalovic
4 min
This paper introduces a clever way to test if large language models (LLMs) can truly capture human personality in generated text. Instead of just asking an LLM to self-report its 'personality' (which can be superficial), the authors use a round-trip evaluation: They feed real psychometric profiles from 290 humans into LLMs to generate first-person life stories, then have independent LLMs read those stories and recover the original personality scores. The key question: Does the generated narrative encode personality robustly enough to decode back to the original traits?
The results are impressive—recovery accuracy hits a mean correlation of r=0.750, close to human test-retest reliability (the 'ceiling' of how consistent human personality scores are over time). This holds across 10 generator LLMs and 3 scorer LLMs from 6 providers, showing it's not model-specific. Why does this matter? It proves LLMs, pretrained on human text, have internalized the deep links between personality and language (the lexical hypothesis), enabling realistic behavioral simulation.
Imagine personality as a compressed code in language. The paradigm tests if LLMs can encode real traits into narratives and others can decode them:
This 'round-trip' avoids self-report biases and tests if conditioning creates psychometrically valid representations of individual differences, not just averages.
This validates immersive persona prompting > basic conditioning, using real data (not synthetic) and HEXACO (including Honesty-Humility).
Builds on:
Implications: LLMs can simulate individuals faithfully for research (e.g., therapy, hiring sims), but biases persist. Extends to non-Big Five traits. Future: Longer contexts, multimodality.
In short, this shows pretraining captures personality-language links robustly, enabling 'others' to tell 'your life story' indistinguishably from you.
Personality traits are richly encoded in natural language, and large language models (LLMs) trained on human text can simulate personality when conditioned on persona descriptions. However, existing evaluations rely predominantly on questionnaire self-report by the conditioned model, are limited in architectural diversity, and rarely use real human psychometric data. Without addressing these limitations, it remains unclear whether personality conditioning produces psychometrically informative representations of individual differences or merely superficial alignment with trait descriptors. To test how robustly LLMs can encode personality into extended text, we condition LLMs on real psychometric profiles from 290 participants to generate first-person life story narratives, and then task independent LLMs to recover personality scores from those narratives alone. We show that personality scores can be recovered from the generated narratives at levels approaching human test-retest reliability (mean r = 0.750, 85% of the human ceiling), and that recovery is robust across 10 LLM narrative generators and 3 LLM personality scorers spanning 6 providers. Decomposing systematic biases reveals that scoring models achieve their accuracy while counteracting alignment-induced defaults. Content analysis of the generated narratives shows that personality conditioning produces behaviourally differentiated text: nine of ten coded features correlate significantly with the same features in participants' real conversations, and personality-driven emotional reactivity patterns in narratives replicate in real conversational data. These findings provide evidence that the personality-language relationship captured during pretraining supports robust encoding and decoding of individual differences, including characteristic emotional variability patterns that replicate in real human behaviour.
Alex: A full profile. Do the stories match real people otherwise?
Sam: They checked ten features—like taking charge or showing vulnerability—against the same people's real conversations. Nine out of ten linked up, about twice the random chance level.
Alex: Solid check against superficial faking. To confirm without clues, how did they test matching?
Sam: They hid details, mixed each story with four random profiles, and had AIs match them. Right picks hit 79%—well above chance. Scans of millions of words found no close echoes from tests. Personality emerges through life examples, not parroting.
Alex: With recovery near human levels, where's the signal loss?
Sam: Prompts slightly push traits, like toward honesty. Unguided stories show AI defaults—often agreeable and calm. Subtracting defaults shows conditioning preserves most of the profile. Strongest for extraversion, weaker for carefulness or openness—matching how they show in stories.
Alex: Defaults explain some gap, but conditioning overrides—even across traits?
Sam: Yes. Even extra traits like trust recover well, tapping broad signals from training data. Personality lives in word choices and styles from human texts. Stronger for social traits fits life stories' focus on relationships.
Alex: This shows AI weaves scores into stories another AI recognizes nearly as well as human retests—bridging numbers to behavior, with limits like cultural narrowness, no human checks, and English data biases. Meaningful step in how AI handles traits through language. Thanks, Sam. Thanks for listening to ResearchPod.
Sam: My pleasure, Alex.