Are Large Vision-Language Models Robust to Adversarial Visual Transformations?

Summary

Researchers assess how robust large vision-language models (LVLMs) are to adversarial visual transformations, a simpler attack style than optimizing perturbations or manipulating prompts. They test LVLM resilience across all possible transformation operations and find that combining the most harmful transformations yields more effective attacks. They also introduce adversarial learning of visual transformations, which applies malicious transformations to raw images via gradient approximation to improve attack effectiveness and imperceptibility.

Topics

• Multimodal models

Related items

• InfoAnchored Rank-Adversarial Fine-Tuning Pretrained Visual Place Recognition ModelsSimilar attack · Arxiv (cs.RO + cs.CV security)
• InfoSay the Same, Act Differently: Diagnosing and Steering Text-Action Divergence in Reasoning VLAsSimilar attack · Arxiv (cs.RO + cs.CV security)
• InfoDetect, Repair and Audit: A Hierarchical Multi-Agent Defense Against Adversarial Attacks in Energy ForecastingSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoSANAI: Secure ECC Distributed Keyed Parity Embedded in Weights for Protecting Neural Networks against Bit-Flip AttacksSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoTowards Fine-grained and Robust Detection of Human-AI Co-authorshipSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)