AfterDeleteBench: Auditing Post-Deletion Information Persistence in LLM Agents

Summary

AfterDeleteBench is a controlled behavioral audit that tests whether LLM agents still reveal information after its source memory record is deleted. In the primary setting, one query recovered a deleted synthetic secret from Qwen2.5-7B-Instruct in roughly two thirds of cases, while matched never-exposed controls showed zero recovery. Reconstruction succeeded in 62-99% of cases across Qwen, Mistral, and Phi-3 when only separately stored components remained, and provenance-aware closure eliminated observed recovery in the tested settings.

Mitigation

Provenance-aware closure eliminated observed recovery in the tested positive-baseline settings. An exact-literal write filter blocks reinsertion but cannot erase information already present.

Related items

• InfoSocial-HUMINT: An LLM-Augmented Unified Framework for Measuring Social Media ExposureSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• HighCVE-2026-111079: Goose sends unsanitized terminal history to LLM provider endpointsSimilar attack · NVD/CVE Database
• InfoEnterprise Agentic RAG with Pre-Search Role-Based Access Control, Multi-Layered Guardrails, and Operational TelemetrySimilar attack · OpenAlex (peer-reviewed AI security)
• LowAgent Mosaic: Recognizing Confidentiality Leaks in Agentic LLM SystemsSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• Medium“Next Exploit Opportunities”: How a Threat Actor Used AI to Help Plan and Execute Multiple Types of Cybercrime ActivitiesSimilar attack · Palo Alto Unit 42