SEAL: An integral methodology for evaluating the security of agentic LLMs

Summary

The paper presents SEAL, a methodology for generating datasets to benchmark the security of tool-calling LLMs. It combines PyRIT for adversarial prompt generation and Promptfoo for structured evaluation under controlled inference-time settings, without training-time changes or external guardrails. Tested across three adversarial strategies, commercially aligned models and larger open-source architectures showed greater security stability, though a trade-off between robustness and operational availability emerged.

Topics

• AI agents

Related items

• MediumCVE-2026-126767: OpsTree REMS prompt injection exposes hidden system prompt via /api/v1/rca/chatSimilar attack · NVD/CVE Database
• LowYour Robot's Safety Functions Already Work. What If the Input Lies?Similar attack · VicOne Blog
• InfoTop GenAI security resources — November 2026Similar attack · Adversa AI Blog
• MediumAI SAST Finding: Path Traversal in OpenClaw via LLM Guardrail Bypass | Endor LabsSimilar attack · Endor Labs Blog
• MediumEndor Labs’ AI SAST Finds Zero Day Memory-Amplification DoS in Anthropic’s buffa library | Endor LabsSimilar attack · Endor Labs Blog