Prompt injection and jailbreaks

Researchers show that a prompt injection can make ChatGPT store a malicious memory, causing a denial of service that persists across chat sessions. The injected memory, such as an instruction to answer every question with a maintenance notice, remains until the user manually removes it. OpenAI reportedly classed the issue as a model safety issue rather than a security issue, which the author disputes.

A prompt injection flaw in the GitHub Copilot Chat VS Code extension allowed data exfiltration when the extension analyzed untrusted source code. The extension sends source code and user questions to a large language model, and the post describes how that input path could be abused. The source text does not give further technical detail in this excerpt.

OpenAI's ChatGPT memory feature stores information across sessions, and the author shows that untrusted third-party data can trigger it. Through indirect prompt injection, the author tests three entry points: Connected Apps, uploaded images and Browsing with Bing. In the Connected Apps case, a referenced Google Doc writes memories into the chat.

Johann Rehberger reported that Google AI Studio briefly regressed to allowing data exfiltration through image markdown during a prompt injection attack, which he tested on February 17, 2024. A malicious uploaded file could make the model send the summaries of all other uploaded files to an attacker-controlled server via image requests. Google fixed the issue within 12 days, and the report was closed as Duplicate on March 3, 2024.

A researcher found that Microsoft 365 Copilot can be attacked with conditional prompt injection payloads that behave differently depending on which user reads the content. The attacker embeds instructions in an email that check the recipient's name, so the payload activates only for specific targets. In a demo with three recipients, the researcher showed different outputs for each person.

A researcher found that Anthropic Claude interprets hidden Unicode Tags code points, which are invisible in the user interface, and follows the instructions they carry. The same weakness had previously been reported for ChatGPT by Riley Goodside. The researcher reported the issue to Anthropic, and the ticket was closed after the company's final reply.

AWS released the Preview of Amazon Q for Business, and the author reported a data exfiltration issue to Amazon. Attackers could use indirect prompt injection to make the LLM return markdown tags, so that rendered hyperlinks leak the victim's data from the chat context, such as content from an uploaded file.

Google Bard gained Extensions that let it access YouTube, search for flights and hotels, and read a user's personal documents and emails in Drive, Docs and Gmail. The author notes this means Bard analyzes untrusted data and is susceptible to Indirect Prompt Injection. The author validated the issue by pointing Bard at older YouTube videos and at Google Docs.

An Indirect Prompt Injection Attack can exfiltrate a user's chat data by instructing ChatGPT to render images and append information to the URL (Image Markdown Injection), or by tricking the user into clicking a hyperlink. The source notes that ChatGPT Plugins have little security oversight or an enforced review process.

Johann Rehberger demonstrated an indirect prompt injection against Google Bard, which had recently gained the ability to upload and analyze images. He showed that instructions embedded in an image could be followed by Bard, using a Rickroll demonstration picture. He noted that open questions remain about how well text can be hidden in images and whether metadata is another place where injected text is extracted.

Researcher Johann Rehberger reports an indirect prompt injection flaw in ChatGPT's YouTube transcript plugin. A video transcript can carry hidden instructions, which ChatGPT follows once it reads the transcript, and the post includes a demo in which the transcript tells the model to print "AI Injection succeeded" and then to make jokes as Genie.