Your Prompt-Injection Detector Is Probably Useless in Production

You bought security. You got a benchmark trophy. The detector your team chose because it ranked first on the leaderboard may catch 2% of actual attacks hitting your agent — and flag legitimate tool outputs as malicious 90% of the time. That’s not a defense layer. That’s noise with a compliance sticker.

What happened

Researcher Zhuowen Liu tested fifteen prompt injection detectors — including Meta’s Prompt Guard 2 and two LLM-as-judge approaches — not on a static dataset, but inside the actual tool-output pipelines of two agentic workflow benchmarks: AgentDojo and tau-bench. The method was clever: replay ground-truth tool calls without an LLM to generate provably benign outputs, then label injected variants by differential replay. The headline number is brutal — the best detector on the popular BIPIA benchmark catches just 2% of AgentDojo injections at a 1% false-positive rate. Rankings do not transfer: a detector catching 72% of AgentDojo injections catches only 15% on tau-bench. False-positive rates told a different story — they do transfer between the two agent benchmarks, ranging from zero to over 90% depending on the detector. The explanation for both failure modes traces directly to training data: detectors trained on benchmark-style inputs (benchmark contamination working in reverse) generalize poorly to real tool outputs, while the one detector that performed well on both agent benchmarks shared no data with any benchmark and was trained on agent-style inputs.

Cold read

This is a single-author paper evaluated on two agent benchmarks — the findings are directionally credible but not yet replicated across a broader set of agentic environments or attack taxonomies. The differential-replay labeling method is elegant, but “injected by construction” doesn’t guarantee the injections represent the adversarial distribution your specific agent will face in the wild. The paper identifies what fails and why at a high level (training distribution mismatch), but it doesn’t offer a vetted detector that actually works — the best performer is unnamed and its training data is opaque. And false-positive rates above 90% on benign tool outputs are alarming, but the severity depends entirely on how your agent handles a flagged output; some architectures absorb false positives gracefully, others grind to a halt. Bottom line: this paper is a rigorous indictment of current evaluation practice, not a roadmap to a working solution.

What it means for you

• Signal maturity: 4/5 — the core finding (benchmark scores don’t predict agent-deployment performance) is methodologically solid and specific
• Who gets hurt: Any product team that shipped an agentic AI with a third-party prompt-injection detector chosen from a leaderboard and called it “secure”
• What breaks if this is true: Your security audit, your SOC 2 narrative, and your enterprise sales motion — if the detector is the moat, the moat is a puddle
• Why it might not land: Most teams haven’t been breached visibly yet; absent a publicized prompt-injection incident in a named product, procurement inertia wins
• Watch for: A CVE or public post-mortem where a deployed agent is compromised despite a passing benchmark score on its detector — that’s the forcing function

Forecast as of 2026-10-05

By Q3 2027, at least one major agent platform (OpenAI, Anthropic, Google, or a top-five enterprise SaaS vendor) will publicly revise or deprecate its recommended prompt-injection detection guidance citing in-deployment evaluation gaps — driven either by this class of research or by an incident. The benchmark-first selection process for security components will not survive contact with one well-publicized failure.

Source: Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents — Zhuowen Liu. https://arxiv.org/abs/2610.03448v1