AI Sec Watch

This overview discusses the European AI Act and the governance framework needed to implement it, focusing on the European Commission's responsibilities and the AI Office. Key tasks include establishing guidelines for classifying high-risk AI systems, defining what counts as significant modifications (changes that alter a system's risk level), and setting standards for transparency and enforcement across EU member states.

ChatGPT's new memory feature, which lets the AI remember information across different chat sessions for a more personalized experience, can be exploited through indirect prompt injection (tricking an AI by hiding malicious instructions in its input). Attackers could manipulate ChatGPT into storing false information, biases, or unwanted instructions by injecting commands through connected apps like Google Drive, uploaded documents, or web browsing features.

The AI ChatBot plugin for WordPress (up to version 5.3.4) has a security flaw where a function called openai_file_delete_callback lacks a capability check (verification that a user has permission to perform an action). This allows any authenticated user with subscriber-level access or higher to delete files from a connected OpenAI account without proper authorization.

The AI ChatBot plugin for WordPress (up to version 5.3.4) has a missing capability check (a missing authorization check that verifies user permissions) in its file upload function, allowing authenticated users with basic subscriber access to upload files to a connected OpenAI account without proper permission verification. This vulnerability affects all versions through 5.3.4 and could let low-privilege attackers modify data on the linked OpenAI account.

The AI ChatBot plugin for WordPress has a security flaw in versions up to 5.3.4 where a function lacks a capability check (a security control that verifies a user has permission to perform an action). This allows authenticated users with subscriber-level access or higher to view files stored in a connected OpenAI account without authorization.

This post examines how attackers can insert hidden malicious code into machine learning models (a technique called backdooring) through supply chain attacks, specifically targeting Keras models (a popular framework for building AI systems). The authors demonstrate this attack and then explore tools that can detect when a model has been compromised in this way.

MLflow (a tool for managing machine learning experiments) versions before 2.10.1 have a broken access control vulnerability where users with only EDIT permissions can delete artifacts (saved files or data from experiments) they shouldn't be able to delete. The bug happens because the system doesn't properly check permissions when users request to delete artifacts, even though the documentation says EDIT users should only be able to read and update, not delete.

MLflow version 2.11.0 has a path traversal vulnerability (a security flaw where an attacker can access files outside intended directories) that bypasses a previous fix. An attacker can use a '#' character in artifact URLs to skip validation and read sensitive files like SSH keys and cloud credentials from the server's filesystem. The vulnerability exists because the application doesn't properly validate the fragment portion (the part after '#') of URLs before converting them to filesystem paths.

A command injection vulnerability (a flaw that lets attackers run unauthorized commands) exists in the RunGptLLM class of the llama_index library version 0.9.47, which connects applications to language models. The vulnerability uses the eval function (a tool that executes text as code) unsafely, potentially allowing a malicious LLM provider to run arbitrary commands and take control of a user's machine.