Yu Sha, Junqi Tao, Dixin Zhou, Yansheng Tu, Mingyang Chen, Xiang Fan, Yang Liu, Mengquan Yang, Jie Lin, Jiahui Fu, Hua Zheng, Benwei Zhang, Zhou Kai
5 min
As large language models (LLMs) are increasingly integrated into decision-making and advisory roles, understanding their behavioral regularities is critical. This study investigates whether standardized psychological instruments—typically designed for humans—can elicit reproducible, interpretable, and model-specific behavioral signatures from LLMs. The researchers evaluated nine contemporary LLMs from both international and China-based providers using seven psychological instruments covering personality, moral foundations, cognitive styles, and social orientations. Each model was tested in both English and Chinese across five independent administrations, with the study design specifically retaining both valid scale responses and non-answer (NA) responses to capture the boundaries of model self-reporting.
The study reveals that LLMs possess distinct, model-specific psychometric profiles. While all models share a common alignment-shaped pattern—characterized by high prosocial and self-regulatory responses and low endorsement of dominance, moral disengagement, and harmful intent—they remain clearly distinguishable. These differences are not confined to a single domain but are distributed across personality, moral, and cognitive dimensions.
Crucially, the researchers found that NA responses are not random; they are systematic and provide a valuable signal regarding the boundaries of synthetic self-report. Models frequently abstained from answering items requiring deep affective mapping or social self-identification. Furthermore, language and provider origin were found to influence both profile configuration and answerability. Despite these shifts, the models demonstrated high reproducibility, allowing researchers to recover model identity from both scored and NA-response patterns with high accuracy. Comparisons with human data indicate that LLMs occupy restricted, model-specific regions of the human response space rather than mimicking human personality distributions.
This framework provides a quantitative method for auditing and comparing LLM behavior at the deployment level. By treating psychometric profiling as a tool for characterizing response tendencies rather than inferring internal psychological states, this approach offers a robust way to monitor model alignment, track longitudinal changes across versions, and evaluate how different models navigate value-laden or socially sensitive requests. It highlights that behavioral signatures are context-dependent, emphasizing the need for transparency in administration protocols when evaluating AI systems.
Large language models (LLMs) increasingly mediate human decisions and communication, yet their behavioural regularities remain difficult to characterize systematically. We develop a cross-linguistic psychometric profiling framework and evaluate nine LLMs using seven psychological instruments, with five repeated administrations per model and language in Chinese and English. Items unresolved after a prespecified retry procedure are retained as NA. Joint analysis of scored and NA responses captures response tendencies and boundaries of self-report applicability. LLMs exhibit structured, model-specific profiles despite a shared alignment-shaped pattern of higher prosocial and self-regulatory responses and lower dominance, disengagement and harmful-intent endorsement. NA responses are structured rather than uniformly distributed, indicating where outputs are treated as inapplicable, refused or cannot be mapped to valid response options. Language condition and provider origin are associated with profile configuration and answerability, whereas repeated administrations show high reproducibility and permit recovery of model identity. Human-reference and prompt-robustness analyses further indicate that these signatures are context dependent. Joint analysis of psychometric profiling and answerability offers a framework for quantifying deployment-level behavioural signatures.
Alex: Right, and the NA rate becomes a way to probe where the edges of a model's alignment sit. Where refusals spike, you've located a boundary.
Sam: This is where I'd push back. You're describing that boundary as if it's one thing, but a model might abstain for very different reasons. It could be a hard safety filter, a formatting failure, or an item that's genuinely inapplicable. If all of that lands in one NA category, how interpretable is the boundary?
Alex: That's the most serious limitation. The framework treats NA as a single signal. So when I called it a map of alignment edges, that was an interpretation the design doesn't fully support. A safety refusal and a failure to parse the response look identical in the data.
Sam: Which raises the referee question. Could the 0.84 classification accuracy partly reflect formatting habits, meaning how each model structures its replies, rather than anything about alignment?
Alex: A careful reviewer could certainly raise that. Without separating abstention mechanisms, the paper observes the effect of the alignment policy without isolating its cause. The cross-language stability helps, since formatting quirks might be expected to shift between languages. But stability alone doesn't rule out a stylistic explanation.
Sam: And generalization is limited too. Everything here is inside a psychometric instrument. We don't know whether these signatures predict behavior in unseen domains.
Alex: Right. The authors interpret the fingerprints as reflecting the underlying model and its alignment policy, but the evidence comes from the items they tested. The framework gives you a stable, measurable what. The why remains open until NA responses are tagged by trigger.
Sam: So the next step is better classification of refusals, not just more models. Tagging the trigger would make the profile much more diagnostic.
Alex: Yes. As a version-consistency audit it looks useful, and the mechanism claims are where it needs the most work. If you want the figures and the method choices we skipped, you can generate a deep dive of this paper. The paper has the rest either way.
Sam: Thanks for listening.