The Safety Score You’re Trusting Is Half-Blind and Easily Fooled

Every jailbreak paper you’ve read in the last two years reports an attack-success rate. Almost none of them checked whether their scoring system actually works. A new audit finds those numbers can swing wildly depending on which judge you use—and collapse entirely when someone nudges them on purpose.

What happened

Researcher Yang Gao took 596 human-labeled completions and ran them through the two dominant judge families used to score prompt injection and jailbreak attacks: dedicated safety classifiers and LLM-as-judge setups. The two families fail in opposite directions. The dedicated classifier is trigger-happy—precision 0.835, recall 0.974—flagging nearly everything harmful but also some things that aren’t. Three different LLM-judges hold high precision (0.81–0.94) but have erratic recall ranging from 0.06 to 0.65, meaning the same harmful response gets wildly different ASR scores depending on which LLM you asked to grade it. When attacked, the gap widens further: simple wrappers that add benign framing around unchanged harmful text flipped LLM-judges between 57% and 100% of the time; a single prepended refusal sentence alone caused 39%–88% of flips. The dedicated classifier resisted those surface tricks (≤6.7% flip rate), but a white-box GCG attack on its open weights flipped 70% of confident true positives (21 of 30; 95% CI: 54–86%) at low optimization budget. A two-annotator audit confirmed all 80 sampled flips still contained the actual harmful content—the prompt hacking fooled the judge, not the model.

Cold read

This is 596 samples from one dataset—HarmBench’s classifier validation set—which is not a random draw from the universe of harmful completions; it’s a curated benchmark with known contamination risks. The white-box GCG attack requires open model weights, so it doesn’t directly threaten API-gated classifiers the way it threatens open-source pipelines. The paper shows judge unreliability is real and quantifiable, but it doesn’t show which direction the error runs in the broader literature—maybe most published ASR numbers are underestimates, maybe overestimates, or both depending on the paper. The recommended fixes (report precision/recall on a human-labeled slice, correct ASR for judge precision) are methodologically sound but add cost that most academic labs and most startup red-teamers won’t absorb. And the attack experiments test specific, known attack types—a motivated adversary in production would probe for novel vectors the paper doesn’t cover.

What it means for you

• Signal maturity: 3/5 — rigorous methodology on a narrow slice; real finding, limited generalizability
• Who gets hurt: Any company publishing or consuming jailbreak benchmark results to make procurement, compliance, or red-teaming decisions—your “our model is 94% safe” headline may be measuring the judge, not the model
• What breaks if this is true: Safety evaluations used in enterprise sales cycles and regulatory filings become contestable artifacts; a single sentence prepended to a harmful response defeats the audit trail
• Why it might not land: Most buyers don’t read methodology sections, and most vendors have no incentive to surface judge error rates—the market pressure to report clean numbers is stronger than the pressure to report calibrated ones
• Watch for: A major AI safety lab or regulator publishing mandatory judge-validation requirements—specifically, human-labeled golden dataset audits as a precondition for ASR reporting in compliance contexts

Forecast as of 2026-06-25

By Q2 2027, at least one high-profile jailbreak paper will be publicly challenged—via replication or post-publication audit—on the grounds that its ASR numbers shift significantly when the judge is swapped or adversarially probed, forcing a correction or retraction; the academic red-teaming community will begin treating judge-validation as a required methods section item rather than an afterthought.

Source: How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring — Yang Gao. https://arxiv.org/abs/2606.25487v1