Textual Response-Aware Guidance for Zero-Shot Adversarial Robustness of Vision-Language Models

Summary

Researchers study zero-shot adversarial robustness of vision-language models, which remain vulnerable to adversarial attacks despite strong generalization. They analyze visual representation drift in two existing forms of pre-trained model guidance, feature matching and prediction-distribution matching, and find complementary limitations. They propose Textual Response-Aware Guidance (TRAG), which constrains visual representation drift using the pre-trained text encoder's responses to local input variations, and report state-of-the-art clean and robust performance across architectures and tasks.

Topics

• Adversarial machine learning
• Multimodal models

Related items

• InfoA Layer-wise Study of How Style Rewriting Breaks the Calibration of Zero-Shot LLM Fake-News DetectorsSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoEvaluating the robustness of explainable AI methods to adversarial attacks in intrusion and malware detectionSimilar attack · OpenAlex (peer-reviewed AI security)
• InfoRL-ATTACK: A Generalized Black-Box Adversarial Attack on Language Models via Reinforcement LearningSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoEvaluating Parameter-Efficient Fine-Tuning Attacks Against Non-Transferable Learning DefensesSimilar attack · Elsevier Security Journals
• InfoEscalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier TriageSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)