AI Sec Watch

This is a podcast episode about AI red teaming (simulated attacks to find weaknesses in AI systems) and threat modeling (planning for potential security risks) in machine learning systems. The episode explores how traditional security practices can be combined with machine learning security to better protect AI applications from attacks.

IBM Watson Machine Learning on Cloud Pak for Data versions 4.0 and 4.5 has a vulnerability called SSRF (server-side request forgery, where an attacker tricks the system into making unauthorized network requests on their behalf). An authenticated attacker could exploit this to discover network details or launch other attacks.

MindsDB, a platform for building AI solutions, has a vulnerability in older versions where it unsafely extracts files from remote archives using `tarfile.extractall()` (a Python function that unpacks compressed files). An attacker could exploit this to overwrite any file that the server can access, similar to known attacks called TarSlip or ZipSlip (path traversal attacks, where files are extracted to unexpected locations).

LLM outputs are untrusted and can be manipulated through prompt injection (tricking an AI by hiding instructions in its input), which affects large language models in particular ways. This post addresses how to handle the risks of untrusted output when using AI systems in real applications.

CVE-2023-28312 is an information disclosure vulnerability in Azure Machine Learning, meaning unauthorized people could access sensitive data they shouldn't be able to see. The vulnerability involves improper access control (CWE-284, a weakness where the system doesn't properly check who is allowed to access what), and it was reported by Microsoft.

CVE-2023-29374 is a vulnerability in LangChain versions up to 0.0.131 where the LLMMathChain component is vulnerable to prompt injection attacks (tricking an AI by hiding instructions in its input), allowing attackers to execute arbitrary code through Python's exec method. This is a code execution vulnerability that could allow an attacker to run malicious commands on a system running the affected software.

MindsDB, an open source machine learning platform, has a vulnerability where it unsafely unpacks tar files (compressed archives) using a function that doesn't check if extracted files stay in the intended folder. An attacker could create a malicious tar file with a specially crafted filename (like `../../../../etc/passwd`) that tricks the system into writing files to sensitive system locations, potentially overwriting important system files on the server running MindsDB.

AI prompt injection is a vulnerability where attackers manipulate input given to AI systems, either directly (by controlling parts of the prompt themselves) or indirectly (by embedding malicious instructions in data the AI will later process, like web pages). These attacks can trick AI systems into ignoring their intended instructions and producing harmful, misleading, or inappropriate responses, similar to how SQL injection or cross-site scripting (XSS, a web attack that injects malicious code into websites) compromise other systems.

TensorFlow (an open-source machine learning framework) versions before 2.11.1 have a bug where a malicious invalid input can crash a model and trigger a denial of service attack (making a service unavailable by overwhelming it). The vulnerability exists in the Convolution3DTranspose function, which is commonly used in modern neural networks, and could be exploited if an attacker can send input to this function.