Your AI Agent Registry Is Already Broken at 500 Tools

ByJohn

Your AI Agent Registry Is Already Broken at 500 Tools

You’re probably running in-context routing right now. At 500 agents or tools, it falls apart — accuracy crashes from 85% to 12% as registries scale to thousands. If your multi-agent stack is growing, this paper is describing your near-term production crisis.

What happened

Figure 2: Within-benchmark scaling. Match@1 vs. N N (Left) and pipeline decomposition (Right).
Figure 2: Within-benchmark scaling. Match@1 vs. N N (Left) and pipeline decomposition (Right).

Sorathiya, Zhang, and Akhbari identified a structural failure mode in how multi-agent orchestration systems currently find the right tool, model, or skill: they stuff a registry into the context window and ask an LLM to pick. This works fine at small scale. It collapses catastrophically at large scale. The paper benchmarked the dominant approach — “in-context routing” — across registry sizes from N=10 to N=7,278 capabilities, and found Match@1 (did you get the right answer on the first pick?) falling from 0.85 to 0.12. Their alternative, Enrich-Retrieve-Rank (ERR), does the expensive metadata work offline, then runs a retrieve-then-rank pipeline online without invoking any candidates first. ERR degrades far more gracefully: Match@1 drops from 0.81 to 0.39 at full scale, because the reranker correctly promotes the right capability to first position 70–87% of the time once retrieval surfaces it. The crossover where ERR beats naive in-context routing is approximately N=500 — a threshold many real platforms are already past. Against the strongest baseline (Search&Pick, where the LLM gets a search tool before choosing), ERR leads by 6.5 percentage points on Match@1 at roughly half the cost. Versus the brute-force Full-Ctx baseline, ERR cuts cost by 70x.

Cold read

The paper’s numbers are compelling but the evaluation is self-contained — this is a single team measuring their own production system, and the benchmark details (what constitutes a “match,” how registries were constructed, what query distribution was used) aren’t visible in the abstract, which matters enormously for retrieval precision and recall. Match@1 of 0.39 at 7,278 capabilities sounds like a win over 0.12, but 39% first-pick accuracy is still a lot of misses in production — the paper doesn’t tell us what the retry or fallback cost is when routing fails, which could erode the claimed cost advantages. The “fixed configuration across agent, tool, and skill registries” claim is interesting but unverified by outside parties; a single golden dataset tuned to the authors’ platform could be doing most of the work. The offline enrichment step — turning sparse metadata into “searchable profiles” — is doing significant hidden labor; if your metadata is bad or inconsistent (as it often is in real teams), that step may not produce the gains shown here. Finally, “runs in production” is a reference to one platform — we don’t know its query volume, failure rate, or whether independent teams have replicated the approach.

What it means for you

  • Signal maturity: 3/5 — Production-validated but single-source, with opaque eval details
  • Who gets hurt: Teams building or selling agentic workflow platforms that scale tool/agent catalogs past a few hundred entries — their routing quality silently degrades while they charge customers for reliability
  • What breaks if this is true: The “just add more agents to the registry” growth strategy hits a wall; onboarding new capabilities stops being free and becomes an active accuracy liability
  • Why it might not land: The offline enrichment step requires structured, maintained metadata — most startup registries are a mess of inconsistent descriptions written by different engineers under deadline, which could neutralize the pipeline’s advantages entirely
  • Watch for: Major multi-agent orchestration platform vendors (LangChain, Vertex AI Agent Builder, AWS Bedrock Agents) publishing routing accuracy benchmarks segmented by registry size — silence on this metric after this paper drops is itself a signal

Forecast as of 2026-08-26

By Q2 2027, at least two of the top-five multi-agent platform vendors will publicly acknowledge registry-scale routing degradation and ship a retrieve-then-rank or semantically-indexed discovery layer as a named feature — or face visible customer churn from teams that hit the N=500 wall and blame the platform.

Source: Enrich-Retrieve-Rank: Scaling Capability Discovery Beyond In-Context Routing — Nazib Sorathiya, Daniel Zhang, Bardiya Akhbari. https://arxiv.org/abs/2608.22695v1

Similar Posts