Understanding and Mitigating Role Spoofing in LLMs

Summary

Researchers study role spoofing, where malicious text in tool outputs, such as a webpage an agent visits, impersonates the user by exploiting LLM role confusion. They find that models track who said what using both content cues and structural cues like control tokens and formatting, and that LLMs are highly susceptible to attacks combining fake structure and content. The paper also proposes detecting role spoofing with linear probes trained to predict conversation roles, which it reports is effective across different settings.

Mitigation

Proposed method: detecting role spoofing using linear probes trained to predict roles in a conversation. The source also reports that adversarial training against content-based injections mitigates content-based attacks but leaves models susceptible to structure-based attacks.

Related items

• HighCVE-2026-119050: Agno PandasTools unsafe reflection allows code execution via prompt injectionSimilar attack · NVD/CVE Database
• Low3 Ways AI Agents Already Operate in Your BrowserSimilar attack · BleepingComputer
• InfoAdversa AI Launches Coding Agent Security Platform to Block Cyberattacks on AI Agents in Real TimeSimilar attack · Adversa AI Blog
• InfoSEAL: Sealed Queries for Local Sanitization against Indirect Prompt InjectionSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• MediumNew Prompt Injection Techniques Cover Agent-Planning Attacks Similar attack · CrowdStrike Blog