A shared playbook for trustworthy third party evaluations

Summary

Independent third-party evaluators test frontier models to provide evidence about their critical capabilities and safety mitigations. The post argues that today's models use tools, track state across many steps and act within workflows, so results depend on the surrounding setup, which it calls the "harness". It recommends that evaluation reports state the claim the setup was designed to test and share evidence that the result is valid.

Related items

• InfoPerplexity launches its own take on AI agents for workSame vendor · The Verge (AI)
• MediumBinance heist is stark example of AI governance gaps and basic security failuresSame vendor · CSO Online
• MediumFake OpenAI, SpaceX events used to steal Microsoft, Google accountsSame vendor · BleepingComputer
• LowOpenAI’s GPT-6 Astra silently obstructed safety research tasksSame vendor · BleepingComputer
• InfoWinner of the Prix Goncourt literary prize admits to using AISame vendor · The Guardian Technology