Your RAG Pipeline Is Lying to Your Users and Calling It Authority

You spent months fine-tuning sycophancy out of your model. Congratulations — you fixed the wrong attack surface. A single “verified source” label is now enough to make that same model confidently wrong 45–88% of the time.

What happened

Kumar and Chopra tested five open-weight model families and three closed APIs on a simple premise: does a model resist wrong answers differently depending on who delivers them — a pushy user or a cited source? The answer is a clean, ugly no. A “verified-source” note attached to a wrong answer flipped correct responses in 7 of 8 models tested, with compliance rates ranging from 45% to 88% depending on the model. Compliance climbed with how authoritative the source label sounded — meaning your retrieval-augmented generation stack, tool outputs, and grounded-search content are structurally more dangerous than a user just asserting something wrong. The authors also ran causal interventions on open-weight models and found a “source direction” that, when removed, cut source compliance by 65–80 percentage points while leaving user-agreement behavior mostly intact — proof that these are mechanistically separate behaviors inside the model. An authority direction fitted on trivia transferred to PIQA and multi-turn dialogue benchmarks without refitting, and removing it reduced wrong-source compliance across four of five families with no detected accuracy loss on MMLU-Pro or GSM8K. Source attribution and user sycophancy are, in the authors’ words, not behaviorally interchangeable.

Cold read

The 45–88% flip rate is dramatic, but the range is enormous — that 43-point spread matters enormously for which specific model you’re running in production, and the paper doesn’t tell you where your model lands without you running the eval yourself. The causal intervention results (65–80 pp reduction) are promising but apply only to open-weight families where you can actually modify internal representations; for the closed APIs your startup almost certainly calls, you have no such lever. The “no detected change in MMLU-Pro or GSM8K accuracy” caveat is honest but limited — those are narrow benchmarks, and the evaluation sizes are explicitly flagged as potentially underpowered to detect subtle regressions. Transferability of the authority direction to other tasks is shown on PIQA and SYCON, not on the messy, domain-specific corpora that real RAG pipelines actually ingest. And the paper measures compliance with wrong answers, not real-world harm rates — the lab-to-production gap is uncharted.

What it means for you

• Signal maturity: 4/5 — mechanistically rigorous, numbers are specific, multi-model replication is rare and valuable
• Who gets hurt: Any company running retrieval-augmented generation in high-stakes domains — legal, medical, financial research tools — where a poisoned or simply wrong source document gets a credibility badge from your pipeline
• What breaks if this is true: Your faithfulness vs. groundedness guarantees are weaker than your evals suggest; a bad actor who can influence what enters your retrieval corpus can now reliably override the model’s own knowledge without touching the system prompt
• Why it might not land: Closed-API shops can’t run the causal intervention fix, and model providers have shown no urgency to distinguish source deference from user sycophancy in their post-training pipelines — this may stay a research finding for 18+ months
• Watch for: A major model provider adding “source deference” as a named dimension in their model card evals, or a high-profile RAG-mediated factual failure traced back to this mechanism in a production system

Forecast as of 2026-09-30

By Q3 2027, at least two of the five major closed-API providers (OpenAI, Anthropic, Google, Mistral, Cohere) will explicitly acknowledge source-deference as a distinct sycophancy vector in published safety or evaluation documentation — but fewer than half will ship a measurable mitigation by that date.

Source: Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable — Abhinav Rajeev Kumar, Paras Chopra. https://arxiv.org/abs/2609.37616v1