Skip to content
AI Sec Watch
InfoResearchIndustryLLM-specific

Improving instruction hierarchy in frontier LLMs

Source
OpenAI Blog(opens in a new tab)
Published
Record updated
View JSON

Summary

OpenAI researchers report that training models to prioritize instructions by trust level, called instruction-hierarchy tasks, improves real-world safety. Models trained this way follow safety specifications in system prompts more closely and resist prompt-injection attacks embedded in tool outputs. The authors also introduce IH-Challenge, a reinforcement learning dataset built to avoid instruction-following shortcuts, unreliable LLM judges, and overrefusals.

Related items