Say the Same, Act Differently: Diagnosing and Steering Text-Action Divergence in Reasoning VLAs

Summary

This research shows that visual perturbations can change the physical action of reasoning vision-language-action models while the generated rationale stays word-for-word identical. Using a text-neutral action-subspace (TNAS) diagnostic on Alpamayo (driving) and InstructVLA (manipulation), the authors find that action-sensitive directions are nearly orthogonal to text-loss gradients. In 154 obstacle-annotated driving scenes, TNAS-guided attacks induced a new collision under an identical rationale in 52.6% of scenes under full-image perturbations, compared with 4.55% for unguided action attacks.

Related items

• InfoAnchored Rank-Adversarial Fine-Tuning Pretrained Visual Place Recognition ModelsSimilar attack · Arxiv (cs.RO + cs.CV security)
• InfoDetect, Repair and Audit: A Hierarchical Multi-Agent Defense Against Adversarial Attacks in Energy ForecastingSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoSANAI: Secure ECC Distributed Keyed Parity Embedded in Weights for Protecting Neural Networks against Bit-Flip AttacksSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoTowards Fine-grained and Robust Detection of Human-AI Co-authorshipSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoVulnerabilities and Defenses in Audio-Visual Attacks: A Survey From Audio to Multimodal ModelsSimilar attack · ACM Digital Library (TOPS, DTRAP, CSUR)