Your AI Agent Is Burning Money Every Time It Opens a PDF

Enterprise AI agents are hemorrhaging tokens like a leaky pipe — and the bill lands on your P&L, not the vendor’s. A new paper from Harvard’s data systems group claims to have found a way to make agents get smarter and cheaper at the same time. Before you forward this to your CTO, read the cold water section.

What happened

Researchers identified a structural inefficiency at the core of agentic workflows built on unstructured data: every time an agent answers a question, it re-opens the same documents and burns through massive context windows. On FanOutQA — a benchmark requiring agents to gather evidence across multiple documents — they measured up to 1 million tokens consumed per question when reasoning over raw unstructured sources. Their proposed fix is “agentic data cracking”: when an agent opens a document to answer a query, a parallel sub-agent forks from the already-loaded context at marginal additional cost and extracts structured data likely to serve future queries. This is adaptive (queries determine what gets structured) and speculative (it extracts beyond the immediate question). The headline numbers: pre-structured data is 28× cheaper than raw document reasoning on FanOutQA, and on an extended benchmark with just one related follow-up question per test query, their cracking method cuts total cost by 53% while preserving accuracy — approaching retrieval-augmented generation costs without sacrificing the depth of full agentic reasoning.

Cold read

The 53% cost reduction is real but narrow: it was measured on a benchmark extended by adding one related follow-up question per test question — a controlled, almost idealized scenario for demonstrating cache reuse. Real enterprise workloads are messier; query distributions are fat-tailed, and the “related future query” assumption may not hold in practice at the rate the benchmark implies. The 28× figure compares against an ideal pre-structured store — a hypothetical that doesn’t exist in production — making it a ceiling estimate, not a deployment baseline. The paper is silent on the cost of the cracking sub-agent itself at scale: if speculative extraction misfires often (wrong structure extracted, low reuse rate), you’ve added latency and token spend on top of your existing bill. There’s also a hallucination risk that goes unaddressed in the abstract: structured data extracted by an LLM sub-agent can contain grounding errors that silently propagate into future answers without the document being re-checked. Finally, the benchmark uses FanOutQA — a relatively clean, well-scoped dataset — and there’s no signal on how the method degrades on noisier enterprise corpora like contracts or earnings call transcripts with inconsistent formatting.

What it means for you

• Signal maturity: 2/5 — single benchmark, no production deployment data, no ablation on extraction error rates
• Who gets hurt: vendors selling per-token agentic document processing (legal AI, financial research tools, enterprise search) whose margins depend on customers not noticing token bloat
• What breaks if this is true: the “just throw more context at it” architecture that most enterprise RAG products are currently built on becomes a competitive liability against systems that accumulate structured knowledge over time
• Why it might not land: the speculative extraction bet only pays off if your query distribution has meaningful locality — if every user question is novel, you’re paying for structure nobody reuses
• Watch for: database and data infrastructure players (think DuckDB-adjacent startups or cloud data warehouses) moving to productize “query-driven schema inference” as a first-class feature for LLM pipelines — that’s the sign this idea is crossing from paper to product

Forecast as of 2026-09-01

By Q3 2027, at least two enterprise agentic AI vendors will ship a production feature explicitly marketed as adaptive or speculative document structuring — but independent benchmarks on real customer workloads will show cost reductions closer to 15–25%, not the 53% headline figure from this paper.

Source: Token-Efficient Data Reasoning Agents via Adaptive Structuring of Unstructured Data — Milad Rezaei Hajidehi, Qitong Wang, Stratos Idreos. https://arxiv.org/abs/2608.31082v1