InfoResearchIndustryLLM-specific
Training activation classifiers for safety and alignment
- Published
- Record updated
Summary
OpenAI describes activation classifiers, or probes, that run on the internal activations of its models to detect policy-violating behavior in real time, starting with GPT-5.6. The probes stream decisions that can block a violating turn midway, are described as more robust to jailbreaks than larger text classifiers, and are said to be more efficient to run. Probes must be retrained for each new base model and periodically within a family as weights drift.
Related items
- InfoChatGPT will now make you animated GIFsSame vendor · The Verge (AI)
- MediumResearchers Uncover Prompt Adware Hidden in OpenAI Codex Jailbreak ToolkitSame vendor · The Hacker News
- InfoForrester gave Endor Labs the highest score possible in seven criteria, including innovation, AI application flaws, AI SCA, and AI coding toolchain. | Endor LabsSame vendor · Endor Labs Blog
- InfoUber only fulfilled 4 rides from AI agents last quarterSame vendor · The Verge (AI)
- InfoSam Altman addresses RAM comments: People should ‘put as much as they want in their computers’Same vendor · The Verge (AI)