Your AI Safety Filter Has 25,000 Examples But Only 1,300 That Count

You calibrated your model’s safety threshold on tens of thousands of examples. The math says you’re actually working with a fraction of that. Every conformal predictor, abstention gate, and content filter built on correlated calibration data is operating on a statistical lie — and the damage is invisible until your one deployment goes wrong.

What happened

Adam Noonan formally breaks the foundational assumption underneath a huge swath of deployed ML safety infrastructure. The specific target: systems that set a threshold at a quantile of a calibration set — conformal predictors, safety filters, abstention gates. All of them promise the threshold holds at the stated rate on new data. That promise requires independent calibration examples. Modern pipelines — where examples share a prompt, document, or reasoning trace — violate that assumption structurally. Noonan borrows the survey-statistics concept of “effective sample size” (the field has had this tool since 1965), but proves that for thresholds, the correction is fundamentally different from the correction for averages. The operative quantity is not how correlated the scores are numerically; it’s how often clustered scores land on the same side of the threshold — and that changes with where the threshold is set. The punchline is empirical and brutal: on a released calibration set of 25,028 examples, the effective sample size for threshold reliability measures at roughly 1,300. A 19× haircut.

Cold read

This is a theoretical and measurement paper, not a controlled experiment showing that deployed systems actually fail at higher rates. The 19× shrinkage figure (25,028 → ~1,300) is measured on one specific released calibration set; how badly your calibration set suffers depends entirely on your clustering structure, which Noonan doesn’t characterize for you. The paper says the existing correction in the conformal literature is “the wrong quantity” — a strong claim — but the abstract doesn’t say by how much it errs in practice, only that it can miss in either direction. Crucially, Noonan notes the damage is “invisible in coverage averaged over many runs, and fully felt by whoever deploys once.” That asymmetry is real and important, but it also means the failure mode won’t show up in your standard offline eval pipeline, making it very hard for most teams to even confirm they have this problem before shipping. The closed-form law is proven; whether practitioners can cleanly measure the inputs to that law on their own proprietary datasets is an open engineering question the abstract leaves untouched.

What it means for you

• Signal maturity: 3/5 — Theory is solid and the math is proven; tooling and empirical validation across diverse pipelines is nascent
• Who gets hurt: Any team shipping a conformal predictor, LLM abstention gate, or AI content detection filter calibrated on data grouped by prompt, document, user session, or shared system prompt
• What breaks if this is true: Your stated coverage guarantees (e.g., “our safety filter blocks 99% of harmful outputs”) are miscalibrated by construction; your compliance and legal representations may be wrong
• Why it might not land: Most teams won’t audit their calibration set’s clustering structure because existing eval metrics won’t flag the problem — no red light ever turns on
• Watch for: Conformal prediction libraries (MAPIE, crepes, etc.) shipping an “exceedance design effect” correction as a first-class parameter by mid-2027; if they don’t, the field quietly ignored this

Forecast as of 2026-08-24

By Q3 2027, at least one major ML safety audit framework or conformal prediction library will formally incorporate a threshold-specific clustering correction distinct from the intra-cluster correlation coefficient currently used — or a replication study will publicly demonstrate the gap between stated and realized coverage on a production LLM pipeline, forcing the issue into practitioner awareness.

Source: The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering — Adam Noonan. https://arxiv.org/abs/2608.21262v1