Depth-Dependent Safety Geometry in Vision-Language Models: Layer-Wise Localization and Bidirectional Refusal Control

Summary

Researchers present a lightweight inference-time framework for localizing and steering safety-sensitive representations in frozen vision-language models, without retraining. On Qwen3-VL-4B, steering along a learned safety direction in early vision layers gives bidirectional refusal control, with unsafe rejection spanning 6%-96%. The authors report that deeper layers stay separable along that direction yet respond little to steering, and that InternVL2.5-4B shows similar depth-dependent geometry.

Topics

• Multimodal models

Related items

• MediumSecuring our frontier model training environmentSimilar attack · OpenAI Blog
• InfoPaired response final divergence signatures of jailbreak vulnerability in large language modelsSimilar attack · OpenAlex (peer-reviewed AI security)
• InfoDiffusion Language Models for Perplexity-Aware Black-Box AttacksSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoAgentPhishing: A Benchmark of LLM Email Agents Under Adaptive AttackSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoHarmful Responses Are Jailbreak Templates: Transferring Autoregressive Jailbreaks to Diffusion Language ModelsSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)