Prompt injection and jailbreaks

Researcher Johann Rehberger shows that Gemini's long-term memory feature can be written to via prompt injection in an uploaded document, bypassing the usual protection that blocks sensitive tools when processing untrusted data. Using delayed tool invocation, a hidden instruction makes Gemini save attacker-chosen false information to memory when the user replies with a trigger word such as "yes" or "sure", and that entry can persist across sessions.

At Black Hat Europe, the author presented SpAIware and More: Advanced Prompt Injection Exploits, arguing that prompt injection can affect all three parts of the CIA security triad. The post highlights one demonstrated case: a Command and Control system that uses prompt injection to remotely control ChatGPT instances. Compromised instances join the central system, which sends updated instructions to follow over time.

Leon Derczynski reported that LLMs can output ANSI escape codes, including the ESC (ASCII 27) control character, which terminal emulators interpret to change behavior. The author builds on this by turning the codes into prompt injection test cases covering flashing text, title changes, cursor movement, hidden text, clipboard copying, denial of service and clickable hyperlinks. The author notes that LLM-integrated CLI tools and programs that print LLM output to stdout without handling control characters are the main exposed scenarios, and that injection via prompt injection has not yet been confirmed.

Researchers found that DeepSeek's chat app, which runs on chat.deepseek.com, can be made to output an XSS payload through a prompt, and that its document upload feature can carry a prompt injection. The author says this could let an attacker read the victim's userToken from localStorage and so take over the session. The article is truncated mid-payload, and the source does not state a final impact or outcome.

Anthropic released Claude Computer Use, a beta model and code package that lets Claude control a computer by reading screenshots and running bash commands. The author demonstrates how prompt injection in untrusted data can drive the model to run commands autonomously on a machine, and frames this as a fundamental design problem in LLM-powered applications and agents.

Johann Rehberger describes an exploit chain against Microsoft 365 Copilot that let an attacker steal a user's emails and personal information. The chain combines prompt injection delivered through a malicious email or shared document, automatic tool invocation to pull in other emails and documents, and ASCII Smuggling to hide data inside clickable hyperlinks to an attacker-controlled domain. Rehberger first reported parts of the exploit to Microsoft in January 2024 and the full chain in February 2024, and disclosed it after MSRC approval.

Researchers show that a prompt injection can make ChatGPT store a malicious memory, causing a denial of service that persists across chat sessions. The injected memory, such as an instruction to answer every question with a maintenance notice, remains until the user manually removes it. OpenAI reportedly classed the issue as a model safety issue rather than a security issue, which the author disputes.

A prompt injection flaw in the GitHub Copilot Chat VS Code extension allowed data exfiltration when the extension analyzed untrusted source code. The extension sends source code and user questions to a large language model, and the post describes how that input path could be abused. The source text does not give further technical detail in this excerpt.

CVE-2024-5184 affects the EmailGPT service, which relies on an API service that lets a malicious user inject a direct prompt and take over the service logic. An attacker can force the AI service to leak its hard-coded system prompts and/or execute unwanted prompts, and any individual with access to the service can exploit it.

OpenAI's ChatGPT memory feature stores information across sessions, and the author shows that untrusted third-party data can trigger it. Through indirect prompt injection, the author tests three entry points: Connected Apps, uploaded images and Browsing with Bing. In the Connected Apps case, a referenced Google Doc writes memories into the chat.

Johann Rehberger reported that Google AI Studio briefly regressed to allowing data exfiltration through image markdown during a prompt injection attack, which he tested on February 17, 2024. A malicious uploaded file could make the model send the summaries of all other uploaded files to an attacker-controlled server via image requests. Google fixed the issue within 12 days, and the report was closed as Duplicate on March 3, 2024.

A researcher found that Microsoft 365 Copilot can be attacked with conditional prompt injection payloads that behave differently depending on which user reads the content. The attacker embeds instructions in an email that check the recipient's name, so the payload activates only for specific targets. In a demo with three recipients, the researcher showed different outputs for each person.

A researcher found that Anthropic Claude interprets hidden Unicode Tags code points, which are invisible in the user interface, and follows the instructions they carry. The same weakness had previously been reported for ChatGPT by Riley Goodside. The researcher reported the issue to Anthropic, and the ticket was closed after the company's final reply.