Predicting model behavior before release by simulating deployment

OpenAI developed Deployment Simulation, a method that tests new AI models by replaying real conversations from previous deployments to see how the new model would behave before release. This approach helps identify unexpected problems and predict how often undesired behaviors might occur in real-world use, addressing limitations of traditional evaluation methods like coverage gaps and selection bias (favoring certain test scenarios over others).