Your AI Safety Checklist Is Already Out of Date — And Attackers Know It

Static red teaming is a snapshot; your model keeps changing, and so do the people probing it. A new framework from a Microsoft-affiliated team just made “we ran a red team eval” a dangerously weak excuse — by showing that adaptive, closed-loop testing finds failures that fixed prompt libraries will never catch.

What happened

The team behind CART identified a core flaw in standard automated red teaming: most pipelines replay a fixed seed library of attack prompts, measuring risks you already knew about and missing everything that emerges from the model’s specific failure patterns. CART fixes this by wiring results back into the attack generator — each failure discovered shapes what gets tested next. The system separates three distinct roles: a Challenger (generates attack probes), a Target (the model or agentic workflow under test), and a Judge (evaluates outputs) — keeping these independent so you can study how each role’s choices affect what evidence surfaces. Tested across three evaluation families (Frontier, JAH, and Agentic), CART found more failures and higher average risk scores than static seed replay for every Target that had a comparable baseline. Crucially, the gains held for tool-using agents, suggesting that when your model can call external APIs or browse the web, the attack surface is meaningfully harder to cover with a fixed checklist.

Cold read

The paper’s own authors draw the sharpest limit: “these results describe what the test policies discover, not how often failures occur in real deployments.” That single disclaimer deflates most of the operational urgency. Finding more failures in a controlled red-team harness does not tell you whether those failures are high-probability in production or theoretical edge cases a real user would never construct. The three evaluation families (Frontier, JAH, Agentic) are not defined in the abstract — we don’t know the domain coverage, whether the baselines were strong, or whether “higher average risk” reflects a meaningful severity scale or an LLM-as-judge score with its own calibration problems. The finding that Challenger-Judge choices “affect the evidence uncovered” is honest but also quietly alarming: it means CART’s results are partially a function of who designed the Challenger and Judge, not just the Target model — a source of variance the framework acknowledges but doesn’t fully resolve. No ablations or quantified improvement margins appear in the abstract, so “more failures” is directional, not actionable.

What it means for you

• Signal maturity: 2/5 — Framework is conceptually sound; empirical specifics needed before it changes procurement or compliance decisions
• Who gets hurt: AI product teams that have shipped a one-time red team report to enterprise customers as evidence of ongoing safety diligence — that report is now a liability argument waiting to be made
• What breaks if this is true: “We passed red team eval” becomes as meaningless as “we passed a pen test in 2022” — regulated buyers (finance, healthcare, government) will start demanding continuous, adaptive, and auditable red teaming as a contract term, not a checkbox
• Why it might not land: Running a closed-loop adaptive red team is expensive and slow. Most startups will continue to use static seed libraries because the incremental cost of CART-style continuous testing doesn’t pencil out against the marginal risk reduction — until a regulator or a breach forces the issue
• Watch for: Enterprise AI procurement RFPs adding “continuous red teaming” or “adaptive adversarial evaluation” as a vendor requirement; or a major benchmark contamination incident traced to a model that passed static evals but would have failed adaptive ones

Forecast as of 2026-09-24

By Q3 2027, at least two major enterprise AI compliance frameworks (EU AI Act implementing guidance, NIST AI RMF tooling, or a major cloud provider’s responsible AI certification) will explicitly reference continuous or adaptive red teaming as a recommended practice — but fewer than 20% of AI startups shipping to regulated verticals will have operationalized anything beyond static seed replay.

Source: CART: Closed-Loop Adaptive Red Teaming for Large Language Models — Dongdong Zhang, Tengchao Lv, Yilin Jia, Yuzhong Zhao, Yupan Huang, Wenshan Wu, Xiangyang Zhou, Shaohan Huang, Nan Yang, Li Dong, Lei Cui, Furu Wei. https://arxiv.org/abs/2609.27336v1