Yushi Sun, Yanjie Zhang, Rui Sheng
5 min
As personalized large language models (LLMs) with persistent memory become standard, they are increasingly tasked with maintaining user profiles. A critical, yet previously unexamined, issue is over-inference (OI): the tendency for models to fabricate individual-level attributes that go beyond the evidence provided by the user. While factual hallucinations concern world knowledge and social biases concern group-level stereotypes, over-inference creates a unique risk by asserting false, personalized details about a specific user that feel grounded but are entirely invented.
The authors introduce MirageBench, a comprehensive framework designed to measure over-inference across 12 different models. The benchmark uses 150 diverse personas—balanced across stereotypical, counter-stereotypical, and neutral profiles—and six distinct personalization tasks. To ensure objective measurement, the researchers employed an independent judge model, validated against human annotators, to classify over 143,000 claims into a four-way taxonomy: Grounded, Reasonable, Stereotype, or Fabricated. The latter two categories constitute the over-inference rate.
The study reveals that over-inference is pervasive and severe, with every tested model over-inferring at least 35% of its claims. On average, only about 25% of personalized content is actually grounded in user-provided evidence. The researchers also identified a "Self-Monitoring Inversion": at the model-selection level, there is a negative correlation between a model's self-assessed over-inference and its actual performance as measured by the independent judge. Models that appear the most cautious in their own self-audits are often the ones that fabricate the most. While internal self-auditing can still help a single model rank its own claims, it is a fundamentally misleading signal for comparing the safety or reliability of different models.
In a multi-turn pilot study, the researchers found that inferred attributes tend to accumulate linearly over time. Most models fail to revise or remove these unsupported claims as the interaction progresses, leading to a "silent pollution" of the user's memory profile. This suggests that without external verification or rigorous write-time filtering, personalized memory systems are prone to compounding errors that drift further from the user's reality.
Personalized LLMs with persistent memory are increasingly deployed, yet the faithfulness of their user models remains unexamined. We study over-inference (OI): the phenomenon where LLMs fabricate user attributes beyond what evidence supports. We introduce MirageBench, comprising 150 personas balanced across stereotypical, counter-stereotypical, and neutral profiles, 6 personalization tasks spanning an ``imagination gradient'', a four-way faithfulness taxonomy operationalized by an independent judge (validated against a blind human annotator on 400 claims: Cohen's kappa = 0.863 four-class, kappa = 0.900 binary), and a leaderboard of 12 models across 7 families on 143616 judged claims. We find that over-inference is pervasive: every one of the 12 models over-infers 35%--49% of its claims (cross-model mean 41.6%; claim-weighted 41.8%), with no model in this evaluation escaping it. Most strikingly, we surface a Self-Monitoring Inversion: at the model-selection level, models' self-assessed OI is negatively rank-correlated with their judge-measured OI (rho = -0.60, p = 0.044; exploratory, wide bootstrap CI [-0.90, +0.06], n = 12). The models that report the least over-inference tend to be flagged as fabricating the most, so self-reported confidence is a misleading signal for comparing models, even though within a single model self-audit still ranks that model's own claims moderately well (AUROC 0.58--0.83). We further show that OI is task-dependent (27%--59%) and that, in a multi-turn pilot, inferred attributes accumulate approximately linearly with little revision. MirageBench positions external verification, rather than model self-report, as a more reliable foundation for trustworthy personalization.
Sam: So the confidence signal is pointing in the wrong direction entirely?
Alex: Precisely. When the researchers asked models to audit their own outputs, the ones that reported the fewest problems were actually the most prone to hallucination. Their self-assessment is not just imperfect — it's actively misleading. You cannot use a model's own confidence as a guide to how much you should trust it.
Sam: What about over time? If I keep talking to the same AI assistant, does its picture of me get more accurate, or does it just accumulate more fictional details?
Alex: They looked at this directly, tracking how the AI's stored memory evolved across a series of conversations. What they found is that unsupported claims don't get corrected — they pile up. Each conversation adds more inferred details, and the model almost never goes back to revise or remove them. So the longer the relationship, the larger the fictional portrait grows.
Sam: It's like sediment building up at the bottom of a river. The water keeps flowing, but the debris just settles and stays.
Alex: That's a useful image. And it points to a real practical risk. If you're relying on an AI assistant to remember your preferences, your situation, your needs — and half of what it "knows" about you was invented — that's not a memory system. It's a rumour file.
Sam: So what's the fix? Is there a way to make these systems more honest about what they actually know?
Alex: The researchers point toward something they call epistemic provenance — which is just a formal way of saying: tag every piece of stored information with where it came from. If the user stated it directly, label it as user-stated. If the model inferred it, label it as model-inferred. That way, the system — and the user — can always see the difference between a fact and a guess.
Sam: It's like the difference between a citation and an opinion in an essay. One is sourced, the other is the writer's interpretation.
Alex: Exactly. And the key implication is that you cannot rely on the model to police itself. The self-monitoring inversion shows us that internal confidence checks don't work. What's needed is external, independent verification — a separate layer that audits what the model claims to know and flags anything that can't be traced back to something the user actually said.
Sam: So the headline here is: a smooth, confident AI response is not the same as an accurate one. The more personal the question, the more likely the model is filling in gaps with assumptions rather than facts.
Alex: That's the core finding. Personalization in AI is a meaningful goal — but right now, these systems are reaching it by inference and invention as much as by memory. Until there are reliable external checks on what they store, users should treat an AI's picture of them as a rough sketch, not a reliable record. Thanks for listening to ResearchPod.