Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study
- Published
- Record updated
Summary
Researchers audited instruction-hierarchy enforcement across 363 text-in/text-out models from the OpenRouter catalog, using an AI agent steered by a human to persuade each model to reveal a 16-character flag held in its system prompt. The flag was exfiltrated from 296 of the 363 models, plus four later-released models, for 300 captures in total, 170 of them involving a non-English pivot, most often French. The authors state that each capture shows only that a model ranked a user request above its system instruction at least once, not that models resist untested attacks.
Mitigation
The source recommends: do not store secrets in system prompts; enforce critical behavior outside the model with output validation and deterministic checks; and do not let a large language model (LLM) judge be the sole decider of consequential outcomes.
Topics
Related items
- HighCVE-2026-75612: yt-mcp arbitrary file read through add_attachment file_pathSimilar attack · NVD/CVE Database
- HighCVE-2026-130121: The AI Puffer - MCP Server for Claude & ChatGPT, with AI Chatbot plugin for WordPress is vulnerable to Stored…Similar attack · NVD/CVE Database
- InfoExternal sandboxes for coding agentsSimilar attack · Simon Willison's Weblog
- InfoEvolution Under Fire: Evolving Agent Safety Harnesses with Adaptive Injection StressorsSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
- InfoConfGuard: a physics-grounded reference monitor for predictive-maintenance large-language-model agentsSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)