Your AI Agent Passed Every Check and Still Broke the Law

ByJohn

Your AI Agent Passed Every Check and Still Broke the Law

You bolted guardrails onto every step of your agentic workflow. Compliance signed off. Auditors saw green lights across the board. The agent still violated policy — and no amount of tuning your per-step monitors would have caught it. This is not a bug you can patch. It is a structural hole in how the entire industry has been building governance.

What happened

Figure 2: Authority Creep Violation in the sample under-writing pipeline. Expanding scope at each stage, illustrating how cumulative scope expansion emerges through composition rather than explicit au
Figure 2: Authority Creep Violation in the sample under-writing pipeline. Expanding scope at each stage, illustrating how cumulative scope expansion emerges through composition rather than explicit au

Researchers from (presumably) an enterprise AI context identified and formally defined a class of governance failure they call Compositional Policy Violations (CPVs): situations where every individual step in an agentic workflow passes its own compliance check, yet the complete execution violates the governing policy. The paper is not an empirical benchmark — it is a taxonomy and an architectural argument. The authors define four distinct CPV types: Authority Creep (accumulated permissions exceed sanctioned limits), Threshold Laundering (a limit is circumvented across multiple sub-threshold steps), Cumulative Sum Violation (individually compliant quantities aggregate into a violation), and Context Collapse (information scoped to one context bleeds into another). The core claim is mathematically clean and hard to argue with: a predicate evaluated over a single step cannot, in principle, detect a property that is only determined by the composed sequence. No improvement in per-step monitor accuracy — zero error rate, perfect classifiers — closes this gap. As a remedy, the authors propose a provenance-aware runtime that evaluates policies over complete execution traces, recomputing guarded quantities from raw provenance rather than derived pipeline state. This directly implicates governance stacks built on input-output classifiers, per-turn rails, and span-level evaluators — which is essentially the entire current market for multi-agent orchestration compliance tooling.

Cold read

The paper presents a taxonomy and an architecture sketch, not a measured system. There are no experiments, no false-positive rates, no latency numbers, no production deployments cited in the abstract — so the provenance-aware runtime is currently a design proposal, not a proven artifact. The four CPV categories are intuitively plausible but the authors do not demonstrate that this taxonomy is exhaustive, meaning real deployments may surface failure modes that don’t fit the framework. The argument that step-level monitors are categorically insufficient is logically sound, but the practical severity depends entirely on how often CPVs actually occur in deployed systems — a frequency the paper does not measure. Regulators and enterprise buyers will need case studies with real policy documents and real workflows before this becomes a procurement or audit criterion; a taxonomy paper, however correct, does not move those audiences alone.

What it means for you

  • Signal maturity: 2/5 — Rigorous conceptual framing, zero empirical validation
  • Who gets hurt: Compliance-forward startups selling AI governance tooling built on per-step classifiers or LLM-as-judge spot checks — their product may be architecturally insufficient for regulated use cases
  • What breaks if this is true: Enterprise deals in healthcare, finance, and legal where authority limits and referral thresholds are real regulatory requirements — the entire “we have guardrails” sales narrative collapses if a single CPV audit surfaces
  • Why it might not land: Most current agentic deployments are not yet consequential enough, or not yet audited closely enough, for CPVs to cause observable, attributable harm — the problem is real but the pain point is not yet acute for most buyers
  • Watch for: A regulated-sector incident (a financial workflow that approved cumulative transactions violating per-customer limits, or a healthcare agent that laundered referral thresholds across steps) being publicly attributed to this failure class — that is the forcing function

Forecast as of 2026-09-17

By Q3 2027, at least one major enterprise AI governance vendor (Guardrails AI, Arthur, or a comparable player) will ship a trace-level, provenance-aware compliance evaluation layer explicitly addressing CPV-class failures — either organically or following a documented production incident in a regulated vertical that legacy step-scoped tooling failed to catch.

Source: Compositional Policy Violations: When Step-Level Compliance Fails In Agentic AI Workflows — Ashwini Kurady, Sri Sai Charith Grandhi, Rajesh Gupta, Sumit Mamoria. https://arxiv.org/abs/2609.18820v1

Similar Posts