Skip to content
InfoResearchPeer-reviewedLLM-specific

Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projector

Published
Record updated
View JSON

Summary

Researchers show that black-box targeted attacks on Large Vision-Language Models can be made more precise by working through the projector, a semantic bridge between vision and language. They propose Intermediate Projector Guided Attack (IPGA), which aligns Q-Former query outputs with a target and transfers across models, and IPGA with Residual Query Alignment (IPGA-R), which also preserves non-target content for fine-grained edits. The authors report that IPGA beats baselines on global attacks, IPGA-R wins on fine-grained attacks, and the method transfers to Google Gemini and OpenAI GPT.