Your AI Agent Isn’t Broken — Your Stack Is Lying to You

You benchmarked your agent. It scored near-zero on tool calls. You blamed the model, maybe the prompt, maybe the training data. You were wrong. A single adapter swap moved successful tool executions from zero to 636 on the same model, same weights, same tasks. The model was working. Your measuring instrument wasn’t.

What happened

Researcher Wenbo Wang ran a controlled dissection of agentic workflow evaluation pipelines and found that the “tool-call rate” most teams treat as a model property is actually a property of the model-plus-interface stack. On BFCL v4, holding weights, test cases, decoding, and seeds constant and changing only the serving adapter, the same model scores either 0.00 or 0.96/0.19 — a near-total swing with zero model change. A 2×2 factorial over chat template and parser showed that neither component is defective in isolation; the failure lives entirely in the interaction term, meaning fixing just the template or just the parser buys you exactly nothing. On tau-bench’s 115 interactive retail tasks, the same adapter swap moved server-parsed calls from 0 to 636 and tasks reaching any tool execution from 0 to 103. The silent failure scales up, not away: across a 21× parameter range of Qwen2.5-Coder, the server parses 0/100 calls at every size while well-formed emitted calls reach 80/100 at 32B. Inside a training loop (verl’s AgentLoop at 7B), 45 of 115 generations contained a complete call; zero were accepted, zero executed, zero returned an observation — meaning tool use vs. function calling infrastructure was silently poisoning the RL reward signal. Wang releases a 98-line preflight check claiming to catch every silent failure class identified in the paper.

Cold read

This is a single-author paper on a narrow, reproducible failure mode — not a sweeping claim about model intelligence, and you should not read it as one. The evaluation-time finding is notably anticlimactic: repairing the adapter at inference restored parsing (0→84 calls rescued) but the pass-rate improvement was 53%→62% and explicitly non-significant (small n), so the downstream outcome gains are genuinely unclear. The benchmark contamination angle cuts both ways here — if leading benchmarks have been running misconfigured stacks, scores in the literature may be garbage, but Wang’s own corrected scores don’t prove the models are actually good. The training-loop contamination point (RL reward poisoning) is the most consequential claim and also the least empirically developed in the abstract; it’s asserted from a single model-size/framework combination. And the 98-line preflight check is only as good as its coverage of adapter variants — a claim that needs adversarial testing across the zoo of deployment frameworks before you rely on it.

What it means for you

• Signal maturity: 3/5 — Mechanism is clean and reproducible; downstream business impact is unquantified
• Who gets hurt: Any team running agent evals in-house and using tool-call rate to make hiring, model-selection, or fine-tuning budget decisions — you may be optimizing against noise
• What breaks if this is true: Every leaderboard ranking of agentic models that didn’t control for serving adapter is potentially invalid; vendor “agent benchmark” marketing numbers become actively misleading
• Why it might not land: Most production teams use a single fixed stack and will never hit the interaction failure — it bites hardest at eval time and during RL training, not necessarily at steady-state inference with a stable deployment
• Watch for: Major agentic AI benchmark maintainers (BFCL, tau-bench) issuing adapter-standardization requirements or retracting/revising existing leaderboard entries within 6 months

Forecast as of 2026-09-04

By Q2 2027, at least two of the top-five cited agent benchmarks will publish explicit serving-adapter standardization policies or issue score corrections, citing interface-induced evaluation variance — but the majority of startup agent evals in the wild will still be running without preflight stack validation.

Source: Interface-Induced Trajectory Censoring — Wenbo Wang. https://arxiv.org/abs/2609.03966v1