Simulated Control Evaluations

Summary

Redwood-style researchers released a paper on simulated control evaluations (SCEs), a method for testing whether AI monitors would catch a model deliberately attempting harm. Instead of executing an attacking model's tool calls, an LLM simulator fakes tool outputs and an LLM judge decides whether the attack would have succeeded. In LinuxArena, the gap between simulated and real results was often about as small as the gap between two independent real runs, and the authors say SCEs are faithful where they can check, though extrapolation to deployment involves some uncertainty and long-horizon, multi-agent attacks lack simulators.