Skip to content
AI Sec Watch
InfoResearchPreprint

VagueBench: Benchmarking the Robustness of Embodied Agents on Ambient Context Illusions

Source
Arxiv (cs.RO + cs.CV security)(opens in a new tab)
Published
Record updated
View JSON

Summary

VagueBench is a benchmark built on the AI2-THOR simulator that tests whether embodied agents still complete tasks when instructions omit non-essential context while keeping the task semantics unchanged. The authors name this failure the Ambient Context Illusion, and the benchmark contains 1,303 instruction pairs. Across 6 state-of-the-art agents, task success rate dropped by a relative 19.65% and empty plan rate rose by a relative 54.10%, and the authors report the effect also appears in real-world robotic systems.

Topics