Skip to content
InfoResearchPreprintLLM-specific

Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study

Published
Record updated
View JSON

Summary

Researchers audited instruction-hierarchy enforcement across 363 text-in/text-out models from the OpenRouter catalog, using an AI agent steered by a human to persuade each model to reveal a 16-character flag held in its system prompt. The flag was exfiltrated from 296 of the 363 models, plus four later-released models, for 300 captures in total, 170 of them involving a non-English pivot, most often French. The authors state that each capture shows only that a model ranked a user request above its system instruction at least once, not that models resist untested attacks.

Mitigation

The source recommends: do not store secrets in system prompts; enforce critical behavior outside the model with output validation and deterministic checks; and do not let a large language model (LLM) judge be the sole decider of consequential outcomes.