Skip to content
AI Sec Watch
InfoResearchPreprintLLM-specific

Prompt-Injection Probes Learn the Conversation, Not Just the Injection

Source
Arxiv (cs.CR + cs.CL + cs.LG)(opens in a new tab)
Published
Record updated
View JSON

Summary

Linear probes on hidden states are used to screen agent tool outputs for indirect prompt injection, but they can learn conversation position and text length instead of the attack itself, reaching 0.91-0.98 AUROC without the model. The authors add matched harmless negatives to the training data, cutting false alarms on injection-free conversations from 41-80% to 4-27% across six open models at the default threshold. At a threshold set on normal traffic, the probe still catches a median 95% of attacks that fooled the agent, versus 91% without the new data.

Mitigation

Add matched negatives to the probe's training data: the same conversation without the injection, and harmless text formatted like an injection. The same data also helps a text-only filter.

Topics

Related items