Your LLM Agent Is Burning Money Regenerating Code It Already Wrote Yesterday

Every time your production agent hits the same workflow step, it’s spinning up inference to solve a problem it solved an hour ago. That’s not intelligence — that’s expensive amnesia. A new paper from what appears to be an Amazon team shows a concrete fix, with numbers hard enough to make your cloud bill flinch.

What happened

Researchers built a “tool-making” pipeline that watches an agentic workflow run repeatedly, identifies the procedural steps that get regenerated on every request, and compiles them into versioned, pre-validated tools before deployment. Instead of re-synthesizing code at inference time, the production agent calls the baked tool directly — falling back to live code generation only for novel cases. They tested this on a real fulfillment-center alarm-triage system: an agent diagnosing warehouse alarms against a 44-node standard operating procedure (SOP) across heterogeneous metric backends. Across 1,500 historical alarms, pre-compiled tool calls reduced end-to-end error rate by up to 53% by suppressing run-to-run variance — the hallucination-adjacent problem where repeated identical steps produce different outputs. On latency, tool calls alone cut p50 by 42%; enabling a simplified “direct-call” architecture on top of that — made possible because tools return compact structured verdicts rather than verbose LLM output — cut p50 by a further 62% in a controlled ablation. The pipeline also uses execution traces and live backend schemas as grounding signal during tool synthesis and repair.

Cold read

This is a single-deployment paper from one warehouse alarm-triage system — a domain that is unusually well-suited to this approach because SOPs are explicit, steps are genuinely repetitive, and “correct” is verifiable against labeled historical cases. The 53% error reduction and 42%/62% latency wins are real production numbers, but they don’t tell you how much engineering effort the tool-making pipeline itself consumed, how often synthesized tools required human intervention to repair, or how brittle the versioned tools become when upstream schemas drift (the paper acknowledges drift as an issue but doesn’t quantify its frequency or cost). The “self-evolving” framing in the title is doing serious marketing work: what’s described is a structured pre-compilation loop with human-labeled cases driving repair — not autonomous self-improvement in any open-ended sense. And the fallback to code generation “when needed” is load-bearing; the paper gives no breakdown of how often that fallback fires in production, which is exactly where the old latency and error problems live.

What it means for you

• Signal maturity: 3/5 — Real production numbers, but single-domain, single-deployment evidence
• Who gets hurt: Startups selling “fast agentic workflows” as a feature when they’re just wrapping raw inference with no tool persistence layer — customers with repetitive SOPs will expect this
• What breaks if this is true: The standard advice to throw a bigger model or longer context window at reliability problems gets expensive to defend when pre-compilation cuts errors by half for free
• Why it might not land: Most enterprise agentic deployments don’t have clean labeled historical cases to drive tool repair — without a golden dataset, the synthesis-and-repair loop has nothing to validate against
• Watch for: AWS or another hyperscaler shipping a managed “tool compilation” layer for agent frameworks (LangGraph, Bedrock Agents) within 12 months — this paper has the fingerprints of an internal team productizing something that already works

Forecast as of 2026-07-10

By Q3 2027, at least two major agent orchestration frameworks will ship a native “tool persistence” or “SOP compilation” feature explicitly targeting latency reduction in repetitive workflows — but adoption will stall in enterprises without pre-labeled historical traces, and fewer than 30% of announced deployments will report error-rate improvements exceeding 30%.

Source: Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems — Kalle Kujanpää, Ning Liu, Shahnawaz Alam, Yeshwanth Reddy Sura, Tianyu Yang, Kristina Klinkner, Shervin Malmasi. https://arxiv.org/abs/2607.08010v1