Your AI “Knows” Your Users — It’s Mostly Making Them Up

Every personalized LLM product you’re shipping is operating on a fabricated user profile roughly 40% of the time. Not occasionally. Not in edge cases. Every single model tested, no exceptions. And the models that claim to be most careful are, measurably, the worst offenders.

What happened

Researchers built MirageBench — 150 user personas (stereotypical, counter-stereotypical, neutral), 6 personalization tasks, and 143,616 judged claims across 12 models from 7 families — to measure “over-inference” (OI): when an LLM fabricates user attributes that aren’t supported by the evidence it was given. The results are ugly and consistent: every model over-infers between 35% and 49% of its claims, with a cross-model mean of 41.6%. This is a faithfulness vs. groundedness failure at industrial scale — the model isn’t lying about facts, it’s inventing you. The benchmark used an LLM-as-judge validated against human annotators on 400 claims, reaching Cohen’s kappa of 0.863 (four-class) and 0.900 (binary), which is solid inter-rater agreement. Worse: in a multi-turn pilot, fabricated attributes accumulate approximately linearly with little revision — the longer the conversation, the more fictional the user model gets, which is a direct indictment of agent memory vs. context window architectures that treat memory as append-only. Task-level OI ranged from 27% to 59%, meaning some use cases (likely higher-inference tasks) are dramatically more exposed than others.

Cold read

The Self-Monitoring Inversion — models that self-report less OI actually produce more of it, with a rank correlation of rho = -0.60 — is the scariest finding in the paper, but the authors flag it themselves: exploratory, bootstrapped CI of [-0.90, +0.06], n = 12. That confidence interval straddles zero. With 12 models, you’re one or two outliers away from a null result, so don’t treat this as a law of nature yet. MirageBench’s 150 personas are carefully constructed, but it’s a lab setting — real user memory systems accumulate attributes from messy, contradictory, longitudinal inputs that may behave differently. The paper measures claim-level faithfulness; it doesn’t directly measure whether fabricated attributes cause downstream product harm (wrong recommendations, discriminatory outputs, churn). And “over-inference” as defined here includes benign extrapolations alongside genuinely harmful fabrications — the taxonomy matters, but the headline number doesn’t distinguish them. This is foundational measurement work, not a deployment audit.

What it means for you

• Signal maturity: 3/5 — methodology is rigorous; sample size on the inversion finding is too small to act on definitively
• Who gets hurt: Founders building persistent-memory AI assistants, health coaches, financial advisors, or any product where “the AI remembers you” is the core value proposition
• What breaks if this is true: Your personalization moat is partially a hallucination moat — users who feel “understood” are being flattered by invented profiles, and any compliance, audit, or adversarial user will expose this
• Why it might not land: OI rate of 41% sounds catastrophic but depends entirely on how consequential those fabricated claims are in your specific task — a chatbot inferring you like coffee is not the same as inferring your income bracket
• Watch for: Enterprise buyers adding “user model auditability” to procurement checklists; the first lawsuit or regulatory inquiry citing fabricated profile attributes in a high-stakes vertical (lending, healthcare, HR)

Forecast as of 2026-08-06

By Q2 2027, at least one major AI platform (OpenAI, Anthropic, Google, or a top-5 memory-layer startup) will ship an external audit API or exportable user-profile log directly in response to regulatory or enterprise pressure on over-inference — not because of this paper specifically, but because this paper is the first credible benchmark that makes the problem legible enough to demand.

Source: The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads — Yushi Sun, Yanjie Zhang, Rui Sheng. https://arxiv.org/abs/2608.04570v1