Skip to content
InfoResearchPreprint

Where Is Rank-1 DiM Refusal Intervenable in Multimodal LM Derivatives?

Published
Record updated
View JSON

Summary

This research asks whether causal chat attack lift under a shared mean / rank-1 difference-in-means (DiM) protocol concentrates at the text decoder T or also appears at perceptual encoders, projectors and fusion paths (E/P/F). On HADES adversarial images paired with harmful text and spoken AdvBench, text DiM at T raises attack success rate (ASR) while the same DiM at E/P/F does not. On VLGuard Unsafe prompts (n=200, Llama-Guard) the same T vs. P/E directionality holds, with ASR rising from 0.030 to 0.150 at T.