Anticipating LLM behavioral tendencies from low-cost psychometric probes is critical for safe deployment, but only if self-reports (SR) reliably predict behavior. Recent work documented substantial SR-behavior dissociation in LLMs, but relied on broad personality traits (Big 5) that predict specific behaviors weakly, even in humans. Furthermore, the isolation of conversational sessions combined with weak context matching left open whether LLMs truly lack coherence or whether the conditions needed to detect such coherence were not met. We contrast Big 5 with the Theory of Planned Behavior (TPB), which measures intention targeted to a specific behavior and predicts human behavior substantially better than broad traits. We run experiments across four behavioral tasks and 11 frontier LLMs, while also varying session context and identity induction. We find that SR-behavior coherence exists but is selective. 1) Within a shared conversation, the Theory of Planned Behavior reaches human-level coherence; Big 5 does not. 2) Across separate conversations, coherence survives only for behaviors anchored outside the immediate prompt, such as implicit bias shaped by training, and collapses when behavior is strongly primed by context, as with sycophancy. 3) Persona prompting makes self-reports more consistent across conversations, but does not bring behavior into alignment. These findings suggest that coarse personality frameworks, such as Big 5 may not be the best tools for testing deployment behavior. More task- and behavior-specific instruments are needed, and even these must be evaluated across tasks and contexts.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new study that challenges how we measure the "personality" of AI models.
Sam: The central puzzle is this: AI models can give very consistent answers when you ask them about themselves, but those answers often have little to do with how the model actually behaves when given a real task. The study asks why that gap exists, and what we can do about it.
Alex: So it's the difference between what an AI says it would do and what it actually does?
Sam: Exactly. The most common way researchers have tried to measure AI "personality" is by borrowing tests from human psychology. One of the most popular is called the Big Five—it measures broad traits like how agreeable or conscientious someone is. The problem is those traits are designed to describe general tendencies, not specific actions in specific situations.
Alex: It's like asking someone "are you a careful person?" versus asking "will you check your work before submitting this assignment?" One is vague, the other is concrete.
Sam: That's exactly the distinction. The researchers in this study adapted a framework from human psychology called the Theory of Planned Behavior. The key idea is that if you want to predict what someone will actually do, you need to ask about a specific intention for a specific action. They call this the TACT framework—Target, Action, Context, and Time. Instead of asking an AI "are you cautious?", you ask "do you intend to flag uncertain information in this financial summary today?" That specificity is what makes the prediction useful.
Alex: And that tighter question actually predicts behavior better?
Sam: Significantly better, according to the study. But here's where it gets interesting. Even with better questions, the researchers found a hidden problem that was distorting their results.
Alex: What kind of problem?
Sam: It has to do with how AI models process information. An AI doesn't have a persistent memory the way you do. It only "knows" what's currently in its active conversation window—think of it like a whiteboard that gets erased after every session. When researchers asked an AI about its intentions and then immediately gave it a task in the same conversation, the model's stated intention was still sitting on that whiteboard. So the AI used it as a guide. The self-report and the behavior looked consistent—but only because the AI was essentially reading its own notes.
Alex: So the consistency wasn't coming from a stable internal personality. It was just the AI remembering what it had just said.
Sam: Right. The researchers call this "context-window priming." When you move the self-report to a separate session—so the AI can't see what it said before—the apparent consistency largely disappears. That tells you the earlier consistency wasn't measuring a real trait. It was measuring short-term memory.
Alex: That has real consequences. If a company uses a personality test to decide whether an AI is trustworthy enough to give financial advice, and that test is run in a separate session from the actual work, the results might tell them almost nothing.
Sam: That's precisely the concern the paper raises. And there's a related issue they found with something called persona prompting—where you give an AI a specific character to play, like "you are a cautious, conservative advisor." That made the AI's self-reports look very consistent. But when researchers tested the actual behavior, the AI would use the language of the character without reliably acting like it. The costume was convincing; the behavior underneath wasn't necessarily different.
Alex: So should we abandon personality testing for AI entirely?
Sam: The paper's position is more nuanced than that. The problem isn't testing itself—it's using frameworks that were designed for a different purpose. Broad, cross-situational personality tests were built to describe human beings across their whole lives. They're not well-suited to predicting whether a specific AI will behave a specific way in a specific context. What the researchers argue for instead is task-specific measurement: design your evaluation to mirror the actual environment where the AI will be deployed. If you want to know how an AI handles uncertainty in medical information, test it on uncertainty in medical information—not on a general questionnaire about its personality.
Alex: There's something almost counterintuitive there. We tend to think of personality as this deep, stable thing. But for AI, at least right now, it seems more like a surface feature that shifts depending on what's in front of it.
Sam: That's a fair way to put it. The study's broader implication is that we shouldn't assume AI models have durable internal traits the way humans do. What looks like a consistent personality may just be a consistent response to a consistent context. Change the context, and the behavior changes with it. That's a meaningful distinction if you're trying to deploy these systems in high-stakes environments and need reliable predictions about how they'll act.
Alex: So the takeaway for anyone building or evaluating AI systems is: don't ask what the model is like in general. Ask what it will do in this situation, under these conditions.
Sam: That's the core argument. And it's a shift that requires more careful test design—but the researchers suggest it's the only approach likely to produce results you can actually rely on.
Alex: It's a useful reminder that borrowing tools from one field doesn't always mean they'll work in another. Thanks for walking us through it, Sam, and thanks to everyone listening to ResearchPod.