Patronus: Safeguarding Text-to-Image Models Against Adversarial Fine-Tuning
Summary
Text-to-image models (AI systems that generate pictures from text descriptions) can be tricked by attackers who fine-tune them (adjust their parameters on new data) to bypass safety protections and create unsafe images. This paper introduces Patronus, a defensive framework that makes these models more resistant to such attacks by using a specially trained safety decoder (a component that processes the model's internal representations) that produces corrupted outputs for unsafe content while preserving normal image generation for safe requests.
Solution / Mitigation
The Patronus framework implements two main defenses: (1) a co-trained safety decoder that produces deliberately corrupted output for latent representations (internal data encodings) associated with unsafe content while preserving normal decoding for benign content, and (2) strengthening the decoder and U-Net (the neural network component that generates images) with a non-fine-tunable learning mechanism to resist gradient-based adversarial fine-tuning attacks.
Classification
Affected Vendors
Related Issues
Original source: http://ieeexplore.ieee.org/document/11653447
First tracked: August 24, 2026 at 08:05 PM
Classified by LLM (prompt v3) · confidence: 85%