Your LLM Is Confidently Lying in Your Invoice Pipeline Right Now

ByJohn

Your LLM Is Confidently Lying in Your Invoice Pipeline Right Now

Silent extraction errors in financial and compliance workflows aren’t edge cases — they’re the default failure mode. A single wrong field auto-approved at scale is a reconciliation nightmare or a regulatory event. This paper says every confidence signal you’re currently using to catch those errors is basically useless.

What happened

Figure 2 : Group 4 spatial feature extraction pipeline. Nearby tokens from each call are matched against OCR tokens to construct per-call bounding boxes, from which label neighbourhood match scores, b
Figure 2 : Group 4 spatial feature extraction pipeline. Nearby tokens from each call are matched against OCR tokens to construct per-call bounding boxes, from which label neighbourhood match scores, b

Researcher Nitesh Kumar tested the three most common ways teams validate LLM extraction confidence — token-level log-probabilities, verbalized confidence scores, and multi-sample self-consistency — and found all three “collapse toward all-positive behaviour at practical thresholds.” In plain terms: your model says it’s confident about almost everything, trustworthy or not, so the signal is worthless for triage. The proposed fix, ExtractConf, runs two structurally different LLM calls on the same document: a “Hunter” call extracts fields under schema pressure (prone to hallucination on absent fields), and a “Mapper” call scans holistically (prone to missing visually non-salient fields). Their disagreement becomes a signal. That disagreement gets fused with OCR quality, image quality, and spatial layout into a classifier that needs no domain-specific retraining. On DocILE — a 55-field invoice benchmark with a 26% baseline failure rate — ExtractConf hits 0.928 ROC AUC and cuts selective prediction risk by 70% over the logprob-mean baseline. At 80% coverage (i.e., the model handles 80% of fields automatically), accuracy on that subset reaches 99.1%. Zero-shot prompting transfer to CORD receipts (a different domain entirely) still achieves 0.858 AUC, and lightweight Lasso recalibration drops Expected Calibration Error by 89% and Brier score by 43%.

Cold read

This is a single-author preprint, not peer-reviewed, and the benchmarks are academic datasets — DocILE and CORD — not a live enterprise pipeline with messy PDFs, mixed scan quality, and adversarial edge cases that real AP departments actually see. The 26% failure rate on DocILE is the setup that makes the 70% risk reduction look dramatic; if your actual pipeline has a 5% failure rate, the math on human-review routing looks very different. The cross-domain transfer number (0.858 AUC on CORD) is promising but was boosted by that Lasso recalibration step — meaning you still need labeled calibration data from your target domain to get the best performance, which undercuts the “no retraining” headline. Running two full LLM calls per document instead of one roughly doubles your inference cost and latency on the extraction step, a tradeoff the paper does not price out in any practical sense. The benchmark contamination risk for any LLM-based system evaluated on publicly available invoice datasets is also non-trivial and unaddressed.

What it means for you

  • Signal maturity: 3/5 — Strong benchmark results, but untested on messy real-world pipelines at scale
  • Who gets hurt: Fintech and procurement automation startups currently shipping “99% accurate” extraction claims based on logprob thresholds — that confidence number is largely fictitious
  • What breaks if this is true: The entire “set a confidence threshold and automate” architecture used by most document AI vendors needs a second LLM call baked into the core loop, repricing their cost-per-document economics
  • Why it might not land: Doubling inference calls is a hard sell when margins on document processing are already thin; customers may prefer faster-and-wrong over slower-and-calibrated
  • Watch for: Enterprise document AI players (Hyperscience, Indico, AWS Textract) quietly adding dual-call or disagreement-based routing to their confidence APIs — that’s the tell that this pattern is being validated in production

Forecast as of 2026-06-24

By Q2 2027, at least one of the top-five document AI platforms will ship a publicly documented multi-call disagreement signal as a confidence feature — but fewer than half of mid-market fintech startups will have adopted any form of calibrated confidence routing, because inference cost pressure will win over accuracy improvement at their scale.

Source: Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction — Nitesh Kumar. https://arxiv.org/abs/2606.24420v1

Similar Posts