AI Sec Watch

A researcher discovered that Amazon Q for Business was vulnerable to an indirect prompt injection attack (a technique where an attacker hides malicious instructions in data that gets fed to an AI), which could trick the AI into outputting markdown tags that render as hyperlinks. This allowed attackers to steal sensitive data from victims by embedding malicious links in uploaded files. Amazon identified and fixed the vulnerability after the researcher reported it.

A researcher discovered that LLMs like ChatGPT can be tricked through prompt injection (hiding malicious instructions in input text) by using invisible Unicode characters from the Tags Unicode Block (a section of the Unicode standard containing special code points). The proof-of-concept demonstrated how invisible instructions embedded in pasted text caused ChatGPT to perform unintended actions, such as generating images with DALL-E.

NVIDIA Triton Inference Server for Linux and Windows has a vulnerability (CVE-2023-31036) that occurs when launched with the non-default --model-control explicit option, allowing attackers to use path traversal (exploiting how file paths are handled to access unintended directories) through the model load API. A successful attack could lead to code execution (running unauthorized commands), denial of service (making the system unavailable), privilege escalation (gaining higher access levels), information disclosure (exposing sensitive data), and data tampering (modifying files).

CVE-2023-7215 is a cross-site scripting (XSS) vulnerability, a type of attack where malicious code gets injected into a webpage that a user views in their browser, found in Chanzhaoyu chatgpt-web version 2.11.1. An attacker can exploit this by manipulating the Description argument with malicious image code, and the attack can be performed remotely over the internet. The vulnerability has been publicly disclosed and may already be in use by attackers.

A security researcher presented at the 37th Chaos Communication Congress about Large Language Models Application Security and prompt injection (tricking an AI by hiding instructions in its input). The talk covered security research findings and was made available in video and slide formats for public access.

Gradio is a Python package for building web demos of machine learning models. Versions before 4.11.0 had a file traversal vulnerability (a weakness that lets attackers read files they shouldn't access) in the `/file` route, allowing attackers to view arbitrary files on machines running publicly accessible Gradio apps if they knew the file paths.

CVE-2023-7018 is a deserialization of untrusted data vulnerability (a flaw where an AI library unsafely processes data from untrusted sources) in the Hugging Face Transformers library before version 4.36. This weakness could potentially allow an attacker to execute malicious code through specially crafted input.

OpenAI has begun addressing a data exfiltration vulnerability (where attackers steal user data) in ChatGPT that exploits image markdown rendering during prompt injection attacks (tricking an AI by hiding instructions in its input). The company implemented a client-side validation check called 'url_safe' on the web app that blocks images from suspicious domains, though the fix is incomplete and attackers can still leak small amounts of data through workarounds.

CVE-2023-6730 is a deserialization of untrusted data vulnerability (a security flaw where a program unsafely reconstructs objects from untrusted input, potentially allowing attackers to execute malicious code) found in the Hugging Face Transformers library before version 4.36. The vulnerability has a CVSS score of 4.0, which indicates a moderate severity level (a 0-10 rating of how severe a vulnerability is).