Your RAG Pipeline Is a Full-Table Scan and You’re Paying for It Every Query

Every time a user asks a question, your system is re-reading the entire corpus and throwing the interpretation away. Fifty years ago, databases solved this exact problem with indexes. The paper argues your retrieval-augmented generation stack hasn’t caught up — and your inference bill is the proof.

What happened

Researchers Kyle Wild, Yusuke Takahashi, and Asako Uraki from argue that production RAG systems carry a “hidden interpreter” — on every query, a language model re-derives meaning from raw text, discards that work, and repeats the process on the next query. They propose ingest-time semantic compilation (ISC): compile the corpus’s meaning once at write time into two coupled layers — incrementally maintained embeddings and atomic claims with provenance validated at compile time — then treat that compiled substrate like a database object with schema, maintenance, and migration contracts. Two concrete results back this up. First, incremental updates to the compiled substrate cost 33.7x less than full reconstruction while matching it to floating-point precision. Second, on 500 broadcast-interview transcripts across 32 budget-by-model test cells, compiled claims achieved 85.2% correct answers from ~2,200 reader tokens, versus 72.5% correct from ~16,300 tokens for the best chunk-based configuration. The only baseline that came close — a contextualized-chunk pipeline with hybrid retrieval and reranking — was statistically indistinguishable from ISC but consumed ~21x more query-path tokens to get there. The authors’ interpretation of that parity: the competitive baseline has itself started to compile, implicitly, at query time.

Cold read

The evaluation is a single domain — broadcast-interview transcripts — which is about as structured and clean as corpora get. Whether ISC’s claim-extraction holds up on messier enterprise data (legal contracts, support tickets, code repos, mixed-media dumps) is entirely unproven here. The “atomic claims with provenance validated at compile time” step is doing enormous hidden work: the quality of the entire pipeline depends on how well a model extracts and validates those claims at ingest, and the paper gives no failure-rate numbers for that stage. Factual consistency between source and compiled claim is assumed to be high, not demonstrated under adversarial or ambiguous conditions. The 85.2% vs. 72.5% accuracy gap is real and sizable, but “correct” is evaluated on held-out samples from a domain where ground truth is relatively legible — the metric’s robustness to subjective or multi-hop questions is unknown. Finally, the 33.7x incremental update savings assumes you know your corpus’s read pattern before users arrive; for exploratory or rapidly-shifting use cases, the “write once, read many” premise breaks down.

What it means for you

• Signal maturity: 3/5 — Compelling numbers from a narrow domain; the architecture is sound but real-world validation is thin
• Who gets hurt: RAG-as-a-service vendors and chunking-pipeline tooling companies (LangChain-style orchestration layers, naive vector-DB wrappers) whose moat is query-time complexity, not ingest-time intelligence
• What breaks if this is true: The cost model for inference-heavy RAG products inverts — the expensive work shifts to ingest infrastructure, and per-query margins improve dramatically; companies that priced on query volume suddenly have a structural disadvantage
• Why it might not land: Claim extraction at ingest is itself an LLM call with its own hallucination surface; if the compiled substrate contains bad claims, errors are baked in rather than query-recoverable, making silent failures harder to debug and potentially more dangerous than the current interpreter model
• Watch for: A major vector-database or RAG-infrastructure vendor (Weaviate, Pinecone, LlamaIndex) shipping a native “semantic compilation” ingest layer — that’s the signal this has cleared the productization threshold

Forecast as of 2026-08-24

By Q3 2027, at least one top-five vector database or RAG framework will ship a production feature explicitly marketed as ingest-time claim compilation or equivalent structured-substrate ingest — and at least one enterprise case study will report >20x query-token reduction. If neither happens, the idea stayed academic.

Source: RAG Deserves an Index: Why Ingest-Time Compilation Beats Query-Time Interpretation — Kyle Wild, Yusuke Takahashi, Asako Uraki. https://arxiv.org/abs/2608.20845v1