Your AI Fairness Audit Is Already Expired — And You Don’t Know It

Your AI Fairness Audit Is Already Expired — And You Don’t Know It

You paid for a fairness audit, got a clean bill of health, and kept using the model. The model was quietly updated six months ago. The audit is now a liability, not a shield. And the regulator coming through your door doesn’t care about version history you never kept.

What happened

Researcher Mika Facchini ran a longitudinal fairness audit of large language models from five providers in a venture screening context — essentially testing whether LLMs discriminate when deciding whether to fund startups whose founders differed only by gender, origin, and university. The study ran across 13 months, repeated three times, totaling 248,400 API calls. The core finding: the same model called under the same identifier one year later showed a significantly different bias pattern — same name, different behavior. When a model was retired and replaced by a successor, the successor carried the predecessor’s bias, but in a different size and form. Notably, the bias pattern did not vary by industry sector. Provider notice periods for model changes ranged from zero to six months, meaning you may have zero warning that the system you audited has already been replaced. The paper introduces the concept of “evidence anchoring” — your audit evidence is locked to a specific model snapshot, and when the provider changes the model at their discretion, that evidence expires.

Cold read

This is a single-researcher study in a single task domain (venture screening), which limits how broadly the findings generalize — hiring bias in an HR tool or credit decisioning may drift differently. The paper shows bias changes, but it doesn’t establish whether the direction of change is consistently harmful or whether magnitude increases over time; “significantly different” doesn’t mean “significantly worse.” The model card problem — providers not documenting what changed and when — is real and well-established, but this paper can’t tell us whether re-auditing on a fixed schedule would actually solve the compliance problem or just create audit theater at higher cost. The “evidence anchoring” theory is conceptually tidy but derived from a narrow empirical base; it’s a hypothesis worth regulating around, not a proven mechanism. Five providers over 13 months is not a large sample, and benchmark contamination effects on model behavior over time are not ruled out.

What it means for you

  • Signal maturity: 3/5 — solid empirical observation, weak mechanistic theory, real regulatory exposure
  • Who gets hurt: Any HR-tech, lending, or talent-intelligence startup that completed a third-party fairness audit more than six months ago and hasn’t re-audited
  • What breaks if this is true: Your compliance documentation is a snapshot of a system that no longer exists; regulators or plaintiffs can argue your audit provides no current-state assurance
  • Why it might not land: Regulators may accept point-in-time audits as sufficient standard of care for years yet, especially outside the EU; enforcement timelines are long
  • Watch for: EU AI Act enforcement actions in 2027 that cite stale audits as evidence of insufficient ongoing monitoring — that’s the canary

Forecast as of 2026-11-17

By Q4 2027, at least one EU AI Act supervisory authority will issue formal guidance requiring continuous or triggered re-auditing of high-risk AI systems after provider-side model updates — making point-in-time audit certificates insufficient for compliance purposes in at least one member state.


Source: Auditing a Moving Target: Evidence Anchoring and the Expiry of Fairness Audits of Provider-Served Language Models — Mika Facchini. https://arxiv.org/abs/2611.20901v1

Similar Posts