Your On-Device AI Research Agent Is Lying to You—Here’s the Exact Math

ByJohn

Your On-Device AI Research Agent Is Lying to You—Here’s the Exact Math

A 4-billion-parameter model running on a $1,500 laptop can now write cited research briefs. Founders are already pitching this as “private, local, trustworthy AI.” The paper behind the claim says faithfulness tops out at 0.58 and trustworthy coverage barely clears 0.22. That’s not a product. That’s a liability.

What happened

Figure 2: Two levers on different outcomes, from the eight runs of Table 1 . (a) Faithfulness of cited claims rises with exposure, and the oracle and naive arms converge near 0.58, so faithfulness is
Figure 2: Two levers on different outcomes, from the eight runs of Table 1 . (a) Faithfulness of cited claims rises with exposure, and the oracle and naive arms converge near 0.58, so faithfulness is

A single study locked a 4B model onto a 24 GB laptop and ran it as an on-device research agent—search a corpus, read sources, write a cited brief. The researcher decomposed what everyone else collapses into one score: faithfulness vs. groundedness (does the cited source actually support the claim?) versus trustworthy coverage (does the agent cite the right sources at all?). The key levers were exposure—how many characters of each source the model sees (400 vs. 1,500)—and source quality—gold papers versus retrieved papers. Results: exposure drives faithfulness, lifting it from 0.45 to 0.58 on retrieved sources and from 0.37 to 0.58 on gold sources, with both conditions converging at the higher exposure level. But trustworthy coverage stays stuck near 0.22 regardless of exposure, because retrieval precision and recall is capped at ~0.40—and no amount of extra context fixes which sources get cited. The cost of higher exposure is roughly 235 additional output tokens per source, which is cheap. The practical prescription the paper extracts: fix exposure first, then treat retrieval recall as the only remaining lever worth engineering.

Cold read

This is a single-model, single-hardware, single-corpus study. Generalizing from one 4B generator on one 24 GB laptop to “on-device research agents” as a category is a leap the data cannot support. The LLM-as-judge methodology is internally flagged: the “exact convergence” between exposure conditions holds tightly under the primary judge but is only approximate under the independent second judge—meaning the headline faithfulness finding is partially judge-dependent, not an objective ground truth. A faithfulness score of 0.58 means roughly four in ten cited claims are not supported by the source they cite; the paper frames this as progress, but for any compliance-sensitive or high-stakes use case, it’s a failure rate that ends careers. Trustworthy coverage at 0.22 means the agent is citing the right sources less than a quarter of the time—a number the paper explicitly cannot improve with the levers it tested. Finally, the golden dataset condition (handing the model the correct sources) still only reaches 0.58 faithfulness, which tells you the ceiling isn’t retrieval—it’s the model itself.

What it means for you

  • Signal maturity: 2/5 — single model, single setup, judge-sensitive results; directionally useful, not yet actionable at scale
  • Who gets hurt: Founders building “private on-device research assistants” for legal, medical, or financial workflows who are marketing citation accuracy as a trust signal
  • What breaks if this is true: Any product claiming on-device RAG delivers reliable cited briefs has a 42% per-claim error rate baked in at best; source attribution becomes a legal exposure, not a feature
  • Why it might not land: The retrieval recall ceiling (~0.40) is corpus- and retriever-specific; better embedding models or hybrid search could move that number, potentially unlocking the coverage floor that exposure alone cannot fix
  • Watch for: A replication on 7B–14B models with the same framework—if faithfulness convergence holds but coverage stays stuck near 0.22, the retrieval bottleneck is structural and the “just use a bigger model” escape hatch closes

Forecast as of 2026-07-15

By Q2 2027, at least two on-device agent product launches will publicly revise or quietly remove citation-accuracy claims after user-reported hallucination rates in real deployments surface via independent audits or press coverage—this paper’s 0.22 trustworthy-coverage ceiling is too low to survive contact with professional users.

Source: On-Device Deep Research at 4B: Exposure Bounds Faithfulness, Retrieval Bounds Coverage — Vinay Kumar Chaganti. https://arxiv.org/abs/2607.12257v1

Similar Posts