Your AI Agent Succeeds 77% of the Time and That’s Killing You

You shipped a ReAct agent. Average pass rate looks great in the demo. Then a customer runs the same task five times and gets three different outcomes. That’s not a bug report — that’s a churn event. A new paper puts a number on the chaos, and it’s worse than you thought.

What happened

IBM researchers measured something most teams quietly ignore: not whether an LLM-powered agent succeeds on average, but whether it succeeds every time on the same task. On the AppWorld benchmark, a ReAct-architecture agent running on GPT-4.1 posts a respectable 77% per-run pass rate — but succeeds in all five consecutive runs only 53% of the time. They name this 24-point shortfall the consistency gap. Their fix is a self-evolving framework with two components: a Consistency Analyzer that identifies which steps in an agentic workflow are likely to flip across runs, and a Guideline Generator that converts those fragile steps into targeted rules stored in episodic memory and injected into future executions. On the same benchmark, the framework raises full-success-across-five-runs by +16 points on identical tasks and +13 points on similar-but-different tasks.

Cold read

This is a single benchmark (AppWorld), one agent architecture (ReAct), one model (GPT-4.1) — the result could be specific to that task distribution and tell you little about your customer support bot or your code-generation pipeline. “Five runs” is a thin definition of reliability; production systems face far longer tails, adversarial inputs, and distribution shift over weeks, none of which are tested here. The +16 / +13 point gains are meaningful but the baseline they’re healing is still 69% full-consistency after the fix — you still fail at least once in five tries on roughly 30% of tasks. The self-evolving memory approach also raises a compounding risk the paper doesn’t surface: bad guidelines baked into episodic memory can systematically worsen the agent over time rather than improve it, a failure mode that’s silent and hard to detect. And “similar-task generalization” at +13 points sounds like transfer, but AppWorld’s task similarity is controlled — real-world task drift is messier.

What it means for you

• Signal maturity: 2/5 — single-benchmark academic result, no production validation
• Who gets hurt: Founders selling AI agents on SLA guarantees or “autonomous” workflows — your consistency numbers are probably worse than your demo suggests, and now customers have vocabulary for the complaint
• What breaks if this is true: Any pricing or contract model built around average success rates becomes legally and commercially fragile; buyers will start demanding worst-case guarantees, not mean performance
• Why it might not land: Episodic memory injection increases latency and token cost per run; for high-frequency, low-margin agent tasks the economics may not close, and teams will default to retrying rather than building memory infrastructure
• Watch for: Enterprise RFPs and agent procurement checklists starting to include consistency benchmarks (5-of-5 or N-of-N pass rates) alongside average accuracy — that’s the moment this framing goes from academic to commercial pressure

Forecast as of 2026-09-09

By Q3 2027, at least two major agent evaluation frameworks (e.g., LangSmith, AgentBench successors, or a hyperscaler eval suite) will have added an explicit consistency metric — repeated-run full-success rate — as a standard reported column, directly traceable to this framing of the consistency gap.

Source: Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course — Evelyn Duesterwald, Benjamin Elder, Lilian Ngweta, Shashanka Ubaru, Malgorzata Zimon. https://arxiv.org/abs/2609.08832v1