Textual Response-Aware Guidance for Zero-Shot Adversarial Robustness of Vision-Language Models

Summary

This research examines zero-shot adversarial robustness in vision-language models (VLMs), which remain vulnerable to adversarial attacks. The authors show that two existing forms of pre-trained model guidance, matching visual features and matching prediction distributions, each have limitations that can degrade zero-shot performance. They propose Textual Response-Aware Guidance (TRAG), which constrains visual representation drift using the pre-trained text encoder's responses to local input variations, and report state-of-the-art clean and robust performance across architectures and tasks.

Topics

• Adversarial machine learning
• Multimodal models

Related items

• InfoLiDAR-SHIELD: A Fully Dockerized Adversarial Robustness Assessment Framework for LiDAR-Based 3D Perception ModelsSimilar attack · Arxiv (cs.RO + cs.CV security)
• InfoPASA-KV: Attacking Action-Facing Visual Routes in World Action ModelsSimilar attack · Arxiv (cs.RO + cs.CV security)
• InfoSynIrisAttack: A benchmark dataset and global-local residual modeling framework for synthetic iris attack detectionSimilar attack · Elsevier Security Journals
• InfoFlipJev: Fast Black-Box Red Teaming Finds Universal Adversarial Suffixes for Typed Decision ModelsSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoAdversarial Self-DistillationSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)