Your AI Agent Is Being Hijacked and the Fix Has Been Breaking It

Every tool-using LLM agent you deploy is one malicious webpage away from doing something you never authorized. Worse: the training patches teams have been shipping to fix this have been quietly lobotomizing the agents’ ability to do their actual jobs. A new framework claims to thread that needle — hold your applause.

What happened

Researchers from (among others) Google’s security team identified and then attacked a known weak point: prompt injection attacks embedded in tool outputs — think a malicious string in a web page your agent fetches, a poisoned API response, or a rigged document in a RAG pipeline. Prior training-based defenses did reduce attack success rates, but the paper’s core finding is that they cause “substantial drift in the model’s output distribution” — meaning the defended model behaves differently even in completely benign settings. The specific failure mode they document: a defended agent refuses to act on legitimate instructions when those instructions come from a tool output, because it has been conditioned to distrust that channel. RAISED fixes this by having the model generate its own agentic workflow training scenarios — specifically emphasizing cases where completing a task requires trusting a tool output — and then uses self-distillation to train a student model to match the teacher’s clean-context behavior on both clean and injection-poisoned versions of the same trajectory. The result, per the abstract: substantially reduced attack success rates with utility preserved on both agentic and general-purpose benchmarks, unlike prior defenses.

Cold read

The abstract is light on the numbers that matter most: how much does attack success rate drop, and against what specific injection attacks? “Substantially reduces” without a figure is a flag. The benchmarks used to measure “preserved utility” aren’t named in the abstract, so we can’t assess whether they reflect production agentic behavior or cleaner, more forgiving evals. Self-distillation against self-generated attack scenarios is also a known limitation: the model can only defend against injection styles it imagines, and adversarial injections in the wild will not politely stay in-distribution. This is also a training-time defense, meaning every time you fine-tune, switch base models, or update your tool use stack, you’re back at square one — there is no runtime guarantee. Finally, Google-affiliated authors publishing a defense paper in 2026 should prompt the question: is this designed to generalize, or to work well on whatever internal stack the team had access to?

What it means for you

• Signal maturity: 2/5 — promising direction, zero production validation visible in the abstract
• Who gets hurt: Any founder shipping autonomous agents that ingest external content (browser agents, email assistants, research tools, agentic RAG products) — you are the current threat surface
• What breaks if this is true: The implicit assumption that “we’ll just fine-tune for safety” is free — it isn’t; you may be trading attack resilience for task completion rate, and neither your security team nor your PM is measuring both simultaneously
• Why it might not land: Self-generated training data caps the defense at the model’s own imagination; a creative attacker with knowledge of your agent’s tool schema will find gaps that the model never trained against
• Watch for: A third-party red-team audit (not from the authors) reproducing the attack-success-rate numbers on a publicly named base model with a publicly named benchmark — that’s when this becomes actionable

Forecast as of 2026-10-06

By Q3 2027, at least one major agent framework (LangChain, LlamaIndex, or a hyperscaler’s hosted agent product) will ship a training-based prompt-injection defense citing this line of work — but the default deployment will remain runtime filtering, not fine-tuning, because the retraining cost and capability-regression risk will be too high for most operator teams to absorb.

Source: RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents — Mohamed Dhouib, Clement Elliker, Alexi Canesse, Maël Jenny, Lucas-Andrei Thil, Mahammed El-Sharkawy, Sonia Vanier, Elie Bursztein. https://arxiv.org/abs/2610.06401v1