Ranking Is Not Deciding: Benchmark Shortcuts and Operating-Point Collapse in Query-Time RAG Poisoning Defenses

Summary

This research examines query-time filters that screen retrieved documents to defend against RAG poisoning, showing their reported near-perfect detection depends on how poison sets were built rather than on what the filters read. Removing a single construction cue, a target-question prefix in PoisonedRAG's released code, cut the content filter RAGuard's AUROC from 0.953 to 0.758 and raised its miss rate from 0.191 to 0.763. Padding poison to clean length with word-shuffled filler drove TrustRAG's set-level miss rate from 0.000 to 0.827, while AUROC held. Detection results should be reported with the poison set's construction profile and the false-positive rate at the deployed threshold.

Topics

• Retrieval-augmented generation

Related items

• InfoGuardRAG: A Post-Retrieval Defense against Poisoning Attacks in Retrieval-Augmented GenerationSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoProtection Is Not One Event: A Survey of RAG Protection Research by Measured EndpointSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoPerMIT: Securing Persistent Memory for LLM Agents via Machine-Checked Integrity Theorems and Scalable Community TrustSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoContext-Aware Prompt Injection Detection via Small Language Models in RAG Pipelines: nokast-secureRAGSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoRAGLeak: Supply-Chain Privacy Backdoors in Retrieval-Augmented GenerationSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)