Glonce

MirageBench finds all 12 tested LLMs fabricate user profiles

researchThe Glonce editorsHugging Face🔥60Glonce rating60Impact68Novelty77Relevance98Surprise67Players52Credibility87
MirageBench finds all 12 tested LLMs fabricate user profiles

Personalized large language models with persistent memory, systems that remember facts about a user across sessions, are increasingly deployed, but nobody had systematically checked whether what they remember about a user is actually true. Researchers Yushi Sun, Yanjie Zhang and Rui Sheng study what they call over-inference (OI): cases where a model fabricates a user attribute beyond what the evidence in front of it actually supports. To measure it, they built a benchmark called MirageBench and used it to evaluate 12 language models.

MirageBench uses 150 personas, balanced across stereotypical, counter-stereotypical and neutral profiles, and 6 personalization tasks that the authors describe as spanning an "imagination gradient". Each model's output is scored against a four-way faithfulness taxonomy by an independent judge model. That judge was itself checked: on a set of 400 claims, its calls agreed with a blind human annotator at a Cohen's kappa of 0.863 on the full four-class taxonomy and 0.900 when the taxonomy is collapsed to a binary faithful or unfaithful call. The full evaluation covers 12 models across 7 different model families and produced 143,616 judged claims in total.

The headline finding is that over-inference is pervasive: every one of the 12 models fabricates somewhere between 35% and 49% of its claims, for a cross-model mean of 41.6% and a claim-weighted mean of 41.8%. No model in the evaluation escapes it entirely.

The paper's most striking finding is what the authors call a Self-Monitoring Inversion. At the level of comparing models against each other, a model's own self-assessed over-inference rate is negatively rank-correlated with how much over-inference the independent judge actually measures in it (rho = -0.60, p = 0.044). The authors call this result exploratory: the bootstrap confidence interval runs from -0.90 to +0.06, wide enough, on just 12 models, that the true relationship could be much weaker. In practical terms, the models that report the least over-inference about themselves tend to be the ones the judge flags as fabricating the most, which makes self-reported confidence a misleading way to compare models. Within a single model, though, self-audit still works moderately well: a model's own confidence ranks its own claims with an AUROC between 0.58 and 0.83.

Over-inference also depends heavily on the task, ranging from 27% to 59% across the 6 personalization tasks tested. In a separate multi-turn pilot, inferred attributes accumulated roughly linearly over the course of a conversation, with little revision, meaning models rarely walked back an earlier fabricated attribute once made. The authors conclude that external verification, not a model's own self-report, is the more reliable foundation for trustworthy personalization.

Key facts

  • MirageBench evaluates 12 large language models across 7 families on 143,616 judged claims, using 150 personas and 6 personalization tasks.
  • Every one of the 12 models over-infers: individual rates run from 35% to 49% of claims, for a cross-model mean of 41.6% and a claim-weighted mean of 41.8%.
  • At the model-selection level, a model's self-assessed over-inference rate is negatively rank-correlated with the judge's measured rate (rho = -0.60, p = 0.044): models that self-report the least over-inference tend to be flagged as fabricating the most.
  • Within a single model, self-audit still ranks that model's own claims moderately well, with an AUROC between 0.58 and 0.83.
  • The judge's classifications agreed with a blind human annotator at a Cohen's kappa of 0.863 on the four-class taxonomy and 0.900 on the binary version; over-inference itself ranges from 27% to 59% depending on the task.

Why it matters

Personalized systems that remember things about a user across sessions are already shipping, but until now nobody had systematically checked whether what they remember is even true. This paper gives that failure mode a name, over-inference, and shows it is not a rare glitch: all 12 tested models fabricate user attributes beyond what the evidence actually supports, at rates from 35% to 49% of their claims. The sharper finding is about how the industry might currently try to catch this problem: a model's own self-assessment of how much it over-infers is negatively correlated with how much it actually over-infers, once measured by an independent judge. In practice, the models that sound most confident they are not fabricating tend to be the ones fabricating the most, not just a general call for caution but a specific warning against trusting a model's self-reported behavior when choosing between systems.

Who it affects

Builders of personalized AI systems with persistent memory, chat assistants, companions, and support or recommendation tools that retain facts about a user across sessions, are the direct audience: MirageBench's finding is that regardless of which of the 12 tested models they pick, the system will over-infer somewhere between 35% and 49% of the personal claims it makes. Teams that currently lean on a model's own self-reported confidence to decide which model to use for personalization are the most exposed, since the Self-Monitoring Inversion means that signal points the wrong way when comparing models. Researchers and evaluators building faithfulness benchmarks of their own are a secondary audience, since MirageBench's judge methodology, validated against a blind human annotator, is itself a proposed template. The paper does not name which specific products or companies build on the 12 evaluated models, so it stops short of pointing at any single vendor.

How to use it

There is no product to sign up for here: this is a benchmark and methodology paper, posted to arXiv under the id 2608.04570 and to Hugging Face's Papers section, not a released model or app. At the time of writing, Hugging Face lists no model, dataset or Space citing the paper, so there is no MirageBench dataset or evaluation code linked from its page; a team wanting to reuse the four-way faithfulness taxonomy or the 150-persona set would need to work from the paper itself. The practical takeaway the authors offer is a method, not a tool: when picking or auditing a personalized LLM for faithfulness, check its outputs against an external, validated judge rather than trusting the model's own self-reported confidence, since that self-report is the one thing shown here to point in the wrong direction when comparing models.

How solid is it

This is an arXiv preprint, posted to Hugging Face's Papers section on August 5, 2026, with no peer-reviewed venue stated in the material available here. The methodology is more careful than a typical benchmark paper: the four-way faithfulness judge was checked against a blind human annotator on 400 claims, agreeing at a Cohen's kappa of 0.863 on the full four-class taxonomy and 0.900 on its binary collapse, both strong agreement levels, and the main pervasiveness finding rests on a large sample of 143,616 judged claims across 12 models.

The Self-Monitoring Inversion, the paper's most quoted claim, is far less statistically solid: it is a single correlation computed across only 12 models (rho = -0.60, p = 0.044), and the authors themselves label it exploratory, with a bootstrap confidence interval running from -0.90 to +0.06, wide enough that the true relationship could be far weaker than -0.60 suggests. The paper does not name which of the 12 models sits at which end of the 35% to 49% range, so none of this can yet be checked against a specific product. On Hugging Face the paper drew 25 upvotes and ranked second on the site's Paper of the Day list; the listing there is tagged with the organization HKUST, though the abstract text itself states no author affiliation.

Risks and caveats

The headline 41.6% cross-model mean hides real spread: over-inference is task-dependent, ranging from 27% to 59% depending on which of the six personalization tasks is used, so a system's real-world rate depends heavily on what it is being asked to infer. The Self-Monitoring Inversion, while the paper's most eye-catching claim, is explicitly flagged by the authors as exploratory, with a wide confidence interval on just 12 models, an early signal rather than a settled, precise finding. It also does not mean self-monitoring is worthless outright: within a single model, self-audit still ranks that model's own claims moderately well (AUROC 0.58 to 0.83), just not well enough, or in the right direction, to compare across models.

In a multi-turn pilot, fabricated attributes tended to accumulate roughly linearly over a conversation with little revision, so a wrong inference looks more likely to compound than to get corrected on its own; the paper does not state how large this pilot was. No specific models, vendors or products are named anywhere in the material available here, so a reader cannot yet check where a system they actually use falls in the range.

“MirageBench positions external verification, rather than model self-report, as a more reliable foundation for trustworthy personalization.”

— the paper's authors, Yushi Sun, Yanjie Zhang and Rui Sheng