How enabling two settings tripled our scores on the ARC-AGI-3 benchmark

A team found that GPT-5.6 Sol's performance on ARC-AGI-3, a benchmark testing AI agents' ability to learn and reason about unfamiliar 2D puzzle games, improved dramatically from 13.3% to 38.3% by enabling two API settings: retained reasoning (keeping the AI's internal thoughts between actions) and compaction (a token optimization technique, where tokens are the basic units of text the AI processes). The benchmark's original harness discarded the model's private reasoning after each action and used a rolling truncation window (removing older history as new information arrived), preventing the AI from remembering its past thinking and learning effectively.