AgentCIScope: Grounded Contextual Integrity Evaluation for LLM Agents

Summary

AgentCIScope is a platform and benchmark that tests whether LLM agents preserve contextual integrity when they read and write sensitive data across application environments through tool calls. The authors evaluated 12 agents on 936 executable tasks and found single-run leakage rates from 32.5% to 94.4%, rising by 4 to 19 percentage points when each task is repeated five times. Leakage varies by attribute, with personal preferences at 84.7% and credit-card information at 17.1%.

Related items

• InfoCost-Efficient Privacy Enforcement for Email-Generating LLM Agents under Strategic Information RequestsSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoPrivacy Risks and Leakage Pathways in Agentic and Multi-Agent AI SystemsSimilar attack · ACM Digital Library (TOPS, DTRAP, CSUR)
• LowGHSA-3gh4-cghq-f8v4: Pydantic AI OpenTelemetry instrumentation: retry prompt content is not redacted when `include_content=False`Similar attack · GitHub Advisory Database
• LowGHSA-4x9p-g9wm-8q7f: Pydantic AI OpenTelemetry instrumentation: exception events on tool and agent run spans include content when `include_content=False`Similar attack · GitHub Advisory Database
• LowSeptember 2026 Cyber Threat Landscape: Global Attacks Jump 48% as Phishing and GenAI Data Exposure RiseSimilar attack · Check Point Research