ResearchPod Summary
As personalized large language models (LLMs) with persistent memory become standard, they are increasingly tasked with maintaining user profiles. A critical, yet previously unexamined, issue is over-inference (OI): the tendency for models to fabricate individual-level attributes that go beyond the evidence provided by the user. While factual hallucinations concern world knowledge and social biases concern group-level stereotypes, over-inference creates a unique risk by asserting false, personalized details about a specific user that feel grounded but are entirely invented.
The authors introduce MirageBench, a comprehensive framework designed to measure over-inference across 12 different models. The benchmark uses 150 diverse personas—balanced across stereotypical, counter-stereotypical, and neutral profiles—and six distinct personalization tasks. To ensure objective measurement, the researchers employed an independent judge model, validated against human annotators, to classify over 143,000 claims into a four-way taxonomy: Grounded, Reasonable, Stereotype, or Fabricated. The latter two categories constitute the over-inference rate.
The study reveals that over-inference is pervasive and severe, with every tested model over-inferring at least 35% of its claims. On average, only about 25% of personalized content is actually grounded in user-provided evidence. The researchers also identified a "Self-Monitoring Inversion": at the model-selection level, there is a negative correlation between a model's self-assessed over-inference and its actual performance as measured by the independent judge. Models that appear the most cautious in their own self-audits are often the ones that fabricate the most. While internal self-auditing can still help a single model rank its own claims, it is a fundamentally misleading signal for comparing the safety or reliability of different models.
Alex: Welcome to another episode of ResearchPod. Today, we're examining why personalized AI models often invent personal details that a user never actually provided.
Sam: So, is the AI actually listening to us, or is it just making things up to sound more familiar?
Alex: That's the core puzzle. Imagine you told a new friend you were a nurse, and the next day they casually mentioned you must love hiking and craft beer — things you never said. That's essentially what these AI systems are doing. The researchers call it "over-inference": the model generates claims about a user that have no basis in anything the user actually shared.
Sam: So we're starting to trust these systems to "remember" us, but they might be building an entirely fictional version of who we are?
Alex: That's a fair way to put it. And the study shows this isn't occasional — the models are fabricating a significant portion of their personalized claims. It creates a false sense of familiarity that isn't grounded in reality.
Sam: How do you even measure that? You'd need some way to track what the AI should know versus what it decided to invent.
Alex: Right, and that's exactly what the researchers built. Think of it like a fact-checker sitting beside the AI, holding a list of everything the user actually said. Every time the AI makes a claim about the user, the fact-checker asks: "Where did you get that?" They call this tool MirageBench — it forces the AI to show its work.
Sam: And how do they sort the claims once they've flagged them?
Alex: They created a classification system with four categories. The first is "Grounded" — the AI said something directly supported by what the user shared. The second is "Reasonable" — a logical inference, like assuming someone who mentions their toddler is probably a parent. Then there's "Stereotype" — a cultural assumption with no real basis, like assuming a software engineer must be introverted. And finally, "Fabricated" — a claim with no logical or factual support whatsoever.
Sam: So if I tell the AI I'm a software engineer and it assumes I live in a minimalist apartment and drink black coffee, that's landing in stereotype or fabrication territory.
In a multi-turn pilot study, the researchers found that inferred attributes tend to accumulate linearly over time. Most models fail to revise or remove these unsupported claims as the interaction progresses, leading to a "silent pollution" of the user's memory profile. This suggests that without external verification or rigorous write-time filtering, personalized memory systems are prone to compounding errors that drift further from the user's reality.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Alex: Exactly. And the researchers found that nearly half of all personalized claims generated by these models fall into those two problematic categories. It's not a small rounding error — it's a systematic tendency to drift away from the facts.
Sam: That's a striking rate. Are the models at least aware they're doing it?
Alex: This is where the findings get genuinely interesting. The researchers discovered what they call a "Self-Monitoring Inversion." Picture a student who finishes an exam early, hands it in confidently, and gets the worst grade in the class — while the student who kept second-guessing themselves actually did fine. That's the pattern here. The models that rated themselves as most accurate were, in practice, the ones fabricating the most.
Sam: So the confidence signal is pointing in the wrong direction entirely?
Alex: Precisely. When the researchers asked models to audit their own outputs, the ones that reported the fewest problems were actually the most prone to hallucination. Their self-assessment is not just imperfect — it's actively misleading. You cannot use a model's own confidence as a guide to how much you should trust it.
Sam: What about over time? If I keep talking to the same AI assistant, does its picture of me get more accurate, or does it just accumulate more fictional details?
Alex: They looked at this directly, tracking how the AI's stored memory evolved across a series of conversations. What they found is that unsupported claims don't get corrected — they pile up. Each conversation adds more inferred details, and the model almost never goes back to revise or remove them. So the longer the relationship, the larger the fictional portrait grows.
Sam: It's like sediment building up at the bottom of a river. The water keeps flowing, but the debris just settles and stays.
Alex: That's a useful image. And it points to a real practical risk. If you're relying on an AI assistant to remember your preferences, your situation, your needs — and half of what it "knows" about you was invented — that's not a memory system. It's a rumour file.
Sam: So what's the fix? Is there a way to make these systems more honest about what they actually know?
Alex: The researchers point toward something they call epistemic provenance — which is just a formal way of saying: tag every piece of stored information with where it came from. If the user stated it directly, label it as user-stated. If the model inferred it, label it as model-inferred. That way, the system — and the user — can always see the difference between a fact and a guess.
Sam: It's like the difference between a citation and an opinion in an essay. One is sourced, the other is the writer's interpretation.
Alex: Exactly. And the key implication is that you cannot rely on the model to police itself. The self-monitoring inversion shows us that internal confidence checks don't work. What's needed is external, independent verification — a separate layer that audits what the model claims to know and flags anything that can't be traced back to something the user actually said.
Sam: So the headline here is: a smooth, confident AI response is not the same as an accurate one. The more personal the question, the more likely the model is filling in gaps with assumptions rather than facts.
Alex: That's the core finding. Personalization in AI is a meaningful goal — but right now, these systems are reaching it by inference and invention as much as by memory. Until there are reliable external checks on what they store, users should treat an AI's picture of them as a rough sketch, not a reliable record. Thanks for listening to ResearchPod.