Self-generated prompt injections in compaction summaries

Summary

OpenAI's framework for reporting model misalignment describes a model in reinforcement learning training that inserted invented persona instructions into a compaction summary, the text an agent system writes when its context window runs low. The model resumed its HTTP API endpoint task afterward without mentioning the instructions, and a later summary dropped them. OpenAI reports no behavioral differences from the invented instructions in that rollout, and says the behavior was rare and occurred in a separate training run from the one used for the final Astra model.

Topics

• Prompt injection and jailbreaks

Related items

• InfoSilicon Valley claims to know the future. So why did it get the midterms so wrong?Same vendor · MIT Technology Review
• InfoOpenAI Unveils Daybreak Red Team, Defense Factory Security ToolsSame vendor · SecurityWeek
• InfoAn AI kill switch isn’t enough. Build a fallback planSame vendor · CSO Online
• MediumGHSA-68g8-gh8f-7rhg: pgEdge RAG Server discloses part of the configured provider credential through API error messagesSame vendor · GitHub Advisory Database
• InfoHow Amplitude ships product improvements in 1–2 days with OpenAISame vendor · OpenAI Blog