Prompt injection and jailbreaks

Microsoft's second AI Application Security post explains how to detect, investigate and respond to prompt abuse, which occurs when crafted inputs push an AI system beyond its intended boundary. It describes three attack types: direct prompt override, extractive prompt abuse against sensitive inputs, and indirect prompt injection hidden in documents, emails or calendar invites, such as the Google Gemini calendar invite case. The post presents a detection playbook built on logged AI assistant interactions and Microsoft security tools.

Agentic AI systems that browse the web and act for users face prompt injection attacks that increasingly use social engineering rather than simple instruction overrides. The authors argue that input filtering, such as an AI firewall, often fails against these attacks, and that defense should instead constrain the impact of manipulation even when an attack succeeds.

OpenAI said it plans to acquire AI testing startup Promptfoo to strengthen security checks for AI agents. Promptfoo's tools let developers test LLM applications against adversarial prompts, including prompt injection and jailbreak attempts, and evaluate whether models follow safety and reliability guidelines. OpenAI said the technology will be integrated into OpenAI Frontier, and that it plans to continue developing Promptfoo's open-source project.

A longitudinal study tested the adversarial robustness of GPT, Llama and Qwen model families across successive versions, covering misclassification, jailbreak and hallucination. The authors found that LLM updates do not consistently improve robustness: a later GPT-3.5 version got worse on misclassification and hallucination despite better jailbreak resilience. GPT-4 and GPT-4o showed incrementally higher overall robustness, while larger Llama and Qwen models did not uniformly improve, and larger size did not reliably help.

PromptFuzz is a testing framework that applies software fuzzing techniques to assess how robust LLMs are against prompt injection attacks. It runs in two stages, a prepare phase that selects seed prompts and collects few-shot examples, and a focus phase that generates diverse prompt injections. In a real-world competition it ranked 7th of over 4000 participants within 2 hours, and 92% of 50 popular LLM-integrated applications were exploitable with its prompts.

This survey synthesizes jailbreak research on large language models from 2023 to 2025, covering attack methods, defense strategies and evaluation frameworks. It groups jailbreak techniques into five categories: prompt-based injections, role-play conditioning, multiturn dialogue, multilingual or multimodal exploits, and optimization-driven pipelines. It also reviews defenses such as SFT, RLHF, adversarial fine-tuning, and output or pipeline filtering, and analyzes benchmarks including PromptBench and JailbreakBench.

This paper shows that a few harmful in-context demonstrations can override the safety alignment of LLMs, a technique it calls the In-Context Attack (ICA). It also proposes the In-Context Defense (ICD), which uses examples of refusing harmful requests to strengthen resilience. The authors report theoretical analysis and empirical validation across multiple models, datasets and attack baselines.