Your AI Agent’s “Trust Score” Is a Lie That Will Get Someone Killed

A single number decides whether your autonomous agent acts or stops. That number can be gamed, tuned in hindsight, and — according to new formal analysis — will silently authorize catastrophic actions your policy explicitly forbids. The paper puts a name on the gap, and the gap is everywhere.

What happened

Researcher Serhii Zabolotnii identified what he calls the “assurance-transition gap”: existing benchmarks, audits, and agentic workflow protocols tell you what an agent can do and how to fix it when it fails, but none specify how live, accumulating evidence should change an agent’s operating authority mid-task. His proposed fix is a Runtime Assurance Contract (RAC) — a formal schema that binds autonomy boundaries, evidence state, and non-compensatory gates, meaning a single failed mandatory check blocks action regardless of how well the agent scores overall. The paper then runs a deterministic failure-injection study: 280 constructed cases in agentic coding, tested against a gate-conjunction rule (RAC), a score-only rule, and a restricted-protocol baseline. The score-only rule admitted 80 of 100 block-required injections and all 40 review-required injections — it missed every case where a human should have been in the loop. The paper also proves algebraically that a score-only rule can only match a gate conjunction if and only if the threshold is tuned to not exceed the smallest individual signal weight — a condition that will virtually never hold in production without hindsight fitting. In a prospective synthetic holdout of 24 episodes, two blinded LLM-as-judge evaluators assigned identical labels to all 72 action attempts, with RAC and a stateful baseline both matching those labels.

Cold read

Everything here is synthetic. The paper is explicit: “These studies test mechanisms on synthetic cases; they establish neither deployed safety nor cross-domain effectiveness.” The 280 failure injections were constructed, not drawn from real production logs, which means the failure modes are exactly what the author designed the gate to catch — not the weird, emergent failures that actually kill products. The prospective holdout is 24 episodes evaluated by LLMs grading LLM outputs, which inherits all the reliability problems of benchmark contamination and circular evaluation. The algebraic proof is clean and the formalisms are rigorous, but a formal schema is not an implementation — adopting RAC in a real multi-agent orchestration stack requires someone to define, maintain, and version the mandatory gates, which is an organizational and engineering problem the paper does not solve. The clinical, industrial, and judicial “failure probes” are illustrative vignettes, not empirical studies.

What it means for you

• Signal maturity: 2/5 — Rigorous formalism, zero production validation
• Who gets hurt: Any startup selling AI agents into regulated verticals (healthcare, legal, finance) that is currently using aggregate accuracy scores as the go/no-go gate for autonomous action
• What breaks if this is true: Your compliance story collapses. Regulators and enterprise buyers will eventually demand audit trails showing which specific mandatory checks passed or failed, not an 87% success rate — and your score-based routing has no answer to that question
• Why it might not land: Defining mandatory gates requires domain expertise and legal clarity that most teams don’t have; absent that, RAC degrades into a checklist that gets rubber-stamped just like every other audit artifact
• Watch for: EU AI Act enforcement actions or enterprise procurement RFPs that explicitly require non-compensatory gate logs rather than aggregate performance metrics — that’s the regulatory forcing function that makes this real

Forecast as of 2026-10-01

By Q3 2027, at least one major cloud provider or agent-platform vendor (e.g., AWS, Azure, or a Series B+ agent infrastructure startup) will ship a runtime policy layer with explicit non-compensatory gate semantics, citing regulatory pressure — but fewer than 20% of production agentic deployments will adopt gate-conjunction logic over score-based routing within that window.

Source: Trust Is Not a Score: Runtime Assurance Contracts for High-Risk AI Agents — Serhii Zabolotnii. https://arxiv.org/abs/2609.39717v1