Your Clinical RAG Passes Every Safety Check—and Kills the Wrong Patient

The hallucination detectors are green. The citations are real. The faithfulness score is near-perfect. And the system just presented drug Y’s clinical trial data as evidence for drug X. Meet deceptive grounding: the failure mode your entire eval stack is blind to.

What happened

Researchers at (presumably) clinical AI built a controlled benchmark to stress-test a specific, previously unnamed failure in retrieval-augmented generation: entity attribution failure, which they call deceptive grounding (DG). The setup is precise—a RAG system retrieves real documents about drug Y when the user asked about drug X, generates a response grounded in those real documents, and every automated check (hallucination detector, faithfulness scorer, citation verifier) passes with flying colors because nothing was fabricated. Across 13 models under adversarial conditions, DG rates ranged from 8% to 87%. The kicker: medical and biomedical fine-tuned models hit up to 86.7% DG—domain specialization made the problem worse, not better. In a live deployed system evaluated across 740 drug-disease pairs, the overall DG rate was 7.8%, climbing to 13.6% for recently approved drugs. The authors also show that entity-attribution verification catches DG at 97.0% precision and 98.7% recall—and confirm that no existing evaluation framework implements this check. The gap between faithfulness vs. groundedness is, apparently, a canyon.

Cold read

The 8–87% range is doing a lot of work in the headline—that spread is enormous, and “peak adversarial conditions” is not your production environment. The 7.8% production number is real and meaningful, but we don’t know the retrieval architecture, chunk size, embedding model, or query distribution of the deployed system studied; those variables could swing results dramatically. The ablation finding—that removing entity-specific evidence eliminates DG entirely—is interesting mechanistically, but it also implies that the failure requires a specific retrieval configuration that many simpler RAG systems may not produce by default. The claim that domain fine-tuned models amplify the problem is striking and counterintuitive, but we’re given one data point (86.7%) without knowing which models, how they were fine-tuned, or on what data. Finally, a 97%/98.7% detection rate for their proposed verification layer sounds great, but it’s IPW-adjusted against a human gold standard—real-world clinical deployment adds regulatory, latency, and cost constraints that could make this fix nontrivial to ship.

What it means for you

• Signal maturity: 3/5 — Real finding, real production numbers, but applicability depends heavily on your retrieval setup
• Who gets hurt: Founders selling clinical decision support, drug information platforms, or any medical RAG product that uses “zero hallucinations” as a compliance or sales claim
• What breaks if this is true: Your SOC 2 / FDA 510(k) / CE mark argument that automated eval proves safety is structurally incomplete—regulators who read this paper have a new question your eval stack can’t answer
• Why it might not land: Most non-clinical RAG deployments have lower entity-confusion risk; if your corpus has weak entity overlap between drugs/compounds, DG rates may be negligible in practice
• Watch for: FDA guidance or EU AI Act technical standards starting to require source attribution checks that go beyond faithfulness scoring—this paper gives regulators the vocabulary and the numbers to demand it

Forecast as of 2026-07-13

By Q1 2027, at least one major clinical AI vendor (think Nabla, Abridge, or a comparable player) will publicly add entity-attribution verification as a named eval layer in their safety documentation—either proactively or in response to a compliance inquiry—directly citing this failure mode if not this paper.

Source: Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation — Cedric Caruzzo, Donggeun Yoo, Tae Soo Kim. https://arxiv.org/abs/2607.09349v1