Your AI Due Diligence Agent Just Signed Off on a Lie

The model got the numbers right. It invented the explanation. And it was confident the whole time. If you’re deploying an agentic workflow in any high-stakes document review context — legal, financial, compliance — this paper is the bill coming due.

What happened

Luis M. Sánchez ran a controlled data-room audit of frontier models on document and chart reading tasks — the kind of work sold daily as “AI-powered due diligence.” The core manipulation was simple and brutal: take evidence that models handle fine when it’s easy to find, and bury it. Under buried conditions, accuracy dropped, hallucination increased, forced declarations (the model asserting things it can’t support) increased, tool calls increased, and cost per correct answer increased — all moving in the wrong direction simultaneously. The most damning finding is qualitative but documented: in a production incident, agents paired accurate numeric tables with confidently fabricated structural claims. The paper notes that confidence scores and benchmark calibration did not fully capture these wrong answers, meaning your agentic workflow’s self-reported reliability is not a valid audit trail. Sánchez’s proposed remedy is claim-level receipts — statement-level source attribution rather than answer-level scores — alongside condition-aware scoring and human-adversarial verification. The paper explicitly frames the problem as one of faithfulness vs. groundedness: a response can be fluent, locally accurate on numbers, and still fabricate the interpretive layer that makes those numbers meaningful.

Cold read

This is a single-author audit from one controlled setting — financial due diligence data rooms — and generalizing to your RAG pipeline or your compliance tool is not automatically warranted. The abstract gives us directional findings (accuracy drops, cost rises, fabrication occurs) but publishes no precise accuracy deltas, no model names, no document corpus size, and no statistical significance thresholds you can actually benchmark against. The “production incident” is documented but singular; one incident is not a base rate. The proposed solution — claim-level receipts and adversarial human verification — is operationally expensive and the paper does not demonstrate that it scales, only that it’s necessary in principle. Founders should also note that “buried evidence” is partly a retrieval and context window engineering problem that better RAG architecture may partially mitigate — the paper doesn’t stress-test that alternative.

What it means for you

• Signal maturity: 3/5 — directionally credible, numerically thin for decision-making
• Who gets hurt: Legal-tech, fintech, and govtech founders who’ve shipped agentic document review and are selling accuracy as a feature
• What breaks if this is true: The liability model. If fabricated structural claims accompany accurate tables and confidence scores don’t flag them, the human who signs the memo absorbs 100% of the legal exposure — and they may not know what to verify
• Why it might not land: Better retrieval (hybrid search, re-ranking, citation-enforced prompting) may reduce the buried-evidence failure rate enough that this becomes an engineering problem, not a product-killing one
• Watch for: A due diligence or compliance AI company facing a disclosed error involving correct data paired with wrong interpretation — that’s the canary, and it will be quietly settled unless regulatory disclosure forces it out

Forecast as of 2026-09-15

By Q3 2027, at least one enterprise legal or financial AI vendor will publicly update its accuracy claims or add mandatory human sign-off requirements specifically citing evidence-retrieval reliability failures — either voluntarily under customer pressure or in response to a regulatory inquiry in the EU or US. If no such disclosure occurs, this paper will have described a real risk that the market successfully priced into SLAs without it becoming a public event.

Source: Clean Scores, Buried Evidence, and Confident Wrong: A Receipt-Based Audit of Frontier Agentic QA — Luis M. Sánchez. https://arxiv.org/abs/2609.15319v1