Your AI Gets a Third of Its “Facts” Wrong—and MMLU Never Noticed

Benchmarks said GPT-5-mini and friends were basically solved. Then someone actually read the outputs. The real-world accuracy of LLM parametric knowledge sits at 68.4%—and another 30.5% of claims are so obscure or garbled that the world’s largest encyclopedia can’t even call them right or wrong. If your product ships on model confidence, read this before your users do.

What happened

Researchers at LLMPEDIA took three flagship models—GPT-5-mini, DeepSeek-V3.2, and Llama-3.3-70B—and forced them to generate approximately 1.3 million articles from pure parametric memory, no retrieval-augmented generation allowed. They then audited a stratified sample of atomic claims against Wikipedia and a curated web stack, tagging each claim as supported, refuted, or insufficient-evidence. The uniform random sample accuracy: 68.4%—more than 21 percentage points below what these same model families score on MMLU, the standard flagship benchmark. Nearly a third of claims (30.5%) fell into the “insufficient” bucket: either genuine long-tail knowledge or hallucination that the evidence base couldn’t adjudicate. The authors tie this directly to benchmark contamination logic—MMLU only tests what the experimenter already thought to ask, leaving the rest of the knowledge surface unaudited. The full encyclopedia is live and browsable at a stable URL, with claim-level verdicts and cross-model comparison baked in.

Cold read

The 68.4% figure sounds alarming but carries serious methodological asterisks. Wikipedia-plus-web-stack is not ground truth—it is a coverage-biased proxy that systematically disadvantages niche, recent, or contested knowledge, which means both the “refuted” and “insufficient” buckets are noisy. The “insufficient evidence” category (30.5%) is doing a lot of work here: it collapses genuine long-tail facts the model got right into the same bin as confabulation, which means the real accuracy could be higher or lower—the paper explicitly says the evidence cannot tell. Generating 1.3M articles from parametric memory is also an unusual elicitation method; production use cases involve shorter, prompted outputs where factual consistency dynamics may differ substantially. And MMLU saturation was already a known problem before this paper—this is a well-instrumented restatement of a known critique, not a discovered failure mode.

What it means for you

• Signal maturity: 3/5 — Rigorous instrumentation of a well-known problem, but not yet actionable at the claim level for most builders
• Who gets hurt: Founders selling “AI knowledge base” or automated research tools on the implicit promise that model accuracy ≈ benchmark accuracy; also anyone using LLM-as-judge pipelines where the judge’s own factual recall is in question
• What breaks if this is true: Any SLA or pricing tier built around “accuracy” claims derived from benchmark scores—enterprise buyers will start asking for empirical claim-level audits, not Elo rankings
• Why it might not land: Most production deployments already use RAG or grounding, which sidesteps parametric memory failure almost entirely; operators who’ve already moved off pure parametric retrieval will shrug
• Watch for: Enterprise procurement teams adding parametric-accuracy audits to vendor scorecards, or regulators citing claim-level error rates (not benchmark scores) in AI liability frameworks

Forecast as of 2026-09-02

By Q3 2027, at least one major AI procurement framework (government, healthcare, or legal sector) will cite claim-level factual accuracy—distinct from benchmark scores—as a required vendor disclosure; the LLMPEDIA methodology or a direct derivative will be referenced in that framework. If that doesn’t happen, benchmark scores will remain the de facto standard and this paper becomes a useful citation in academic critique but changes nothing operational.

Source: LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs — Muhammed Saeed, Simon Razniewski. https://arxiv.org/abs/2609.01182v1