Your Expensive Memory Pipeline Is a 3,000× Overpay

A pre-registered study just handed LLM-extraction memory systems their first controlled defeat. Raw conversation turns, ranked by a lightweight typed decision model, matched distilled-fact pipelines at a fraction of the cost — and someone bothered to pre-register it, so you can’t dismiss it as p-hacking.

What happened

Sharma and Lall set up a pre-registered, controlled shootout between two philosophies of agent memory: (1) extract structured facts from conversation history with an LLM, or (2) just select the right raw turns and pass them in. They ran the experiment on two held-out benchmarks — LoCoMo and LongMemEval — using a typed decision model called Jev as the selector. At a tight context window budget on LoCoMo, raw-turn selection proved non-inferior to extraction (one-sided 95% lower bound: −3.0 points against a pre-specified margin of −5 points). Blind human graders agreed with the automated result. The cost gap is the number that should make you wince: raw turns cost 3,061 times less to write than LLM-extracted facts. Budget size matters enormously, though — reranking added 17.4 points on LoCoMo when only 3 of 30 candidates fit, but only 1.5 points at generous budgets, where extraction systems regained accuracy. That one finding probably explains why the published literature has been contradicting itself for two years.

Cold read

Non-inferiority is not superiority. The study shows the methods are close enough, not that selection wins — and the −5-point margin is an assumption the authors chose, not a law of nature. Both benchmarks (LoCoMo, LongMemEval) are written-conversation datasets; how well this transfers to messier real-world agent memory — tool calls, multi-modal inputs, long task threads — is entirely untested. Jev is a typed decision model you’ve probably never heard of and may not be able to drop into your stack tomorrow. The result that reranking lowers correct abstention is buried in the abstract and never really explained — that’s a hallucination and faithfulness risk that deserves far more scrutiny before you rip out your extraction layer. One pre-registered study is a green shoot, not a paradigm shift.

What it means for you

• Signal maturity: 2.5/5 — Pre-registration is credible; single study on two benchmarks is not.
• Who gets hurt: Vendors selling LLM-extraction memory middleware as a premium layer (and the startups who bought annual contracts for it).
• What breaks if this is true: The “distilled facts” moat collapses. If selection from raw turns is good enough, retrieval-augmented generation pipelines simplify dramatically and the engineering surface shrinks.
• Why it might not land: Budget regime determines which approach wins. Enterprise customers with fat context budgets will see extraction pull ahead again, per the paper’s own data.
• Watch for: A replication on agentic, tool-use-heavy corpora (not chat logs). If selection holds there, it’s real. If it doesn’t, this stays a conversational-AI curiosity.

Forecast as of 2026-09-30

By Q3 2027, at least two well-resourced agentic workflow platforms (e.g., LangChain, LlamaIndex, or a major cloud provider’s agent SDK) will ship a “selection-first, extract-on-overflow” memory mode as a default option, explicitly citing cost-parity results like this one — but extraction will remain the default for enterprise tiers.

Source: When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model — Rishabh Sharma, Rishika Lall. https://arxiv.org/abs/2609.34227v1