AI Sec Watch

The EU AI Act is a regulatory framework that requires companies to comply with rules for different types of AI systems on specific timelines, starting with prohibitions on the riskiest AI uses within 6 months and expanding to cover high-risk AI systems (such as those used in law enforcement, hiring, or education) by 24 months after the law takes effect. The article outlines key compliance deadlines, secondary laws the EU Commission might create to clarify the rules, and guidance documents to help organizations understand how to follow the AI Act.

ZenML Server in the ZenML machine learning package before version 0.46.7 has a remote privilege escalation vulnerability (CVE-2024-25723), meaning an attacker can gain higher-level access to the system from a distance. The flaw exists in a REST API endpoint (a web-based interface for requests) that activates user accounts, because it only requires a valid username and new password to change account settings, without proper access controls checking who should be allowed to do this.

The EU AI Act classifies AI systems by risk level, from prohibited (like social scoring systems that manipulate behavior) to minimal risk (unregulated). High-risk AI systems, such as those used in critical decisions affecting people's lives, face strict regulations requiring developers to provide documentation, conduct testing, and monitor for problems. General-purpose AI (large language models that can do many tasks) have lighter requirements unless they present systemic risk, in which case developers must test them against adversarial attacks (attempts to trick or break them) and report serious incidents.

CVE-2024-27444 is a vulnerability in LangChain Experimental (a Python library for building AI applications) before version 0.1.8 that allows attackers to bypass a previous security fix and run arbitrary code (malicious commands they choose) by using Python's special attributes like __import__ and __globals__, which were not blocked by the pal_chain/base.py security checks.

MLflow, a machine learning platform, has a vulnerability where it doesn't properly clean user input from dataset tables, allowing XSS (cross-site scripting, where attackers inject malicious code into web pages). When someone runs a recipe using an untrusted dataset in Jupyter Notebook, this can lead to RCE (remote code execution, where an attacker can run commands on the user's computer).

MLflow has a vulnerability (CVE-2024-27132) where template variables are not properly sanitized, allowing XSS (cross-site scripting, where malicious code runs in a user's browser) when running an untrusted recipe in Jupyter Notebook. This can lead to client-side RCE (remote code execution, where an attacker can run commands on the user's computer) through insufficient input cleaning.

ONNX (a machine learning model format library) versions 1.15.0 and earlier have an out-of-bounds read vulnerability (accessing memory outside intended boundaries) caused by an off-by-one error in the ONNX_ASSERT and ONNX_ASSERTM functions, which handle string copying. This flaw could allow attackers to read sensitive data from memory.

ONNX (a machine learning model format) versions 1.15.0 and earlier contain a directory traversal vulnerability (a security flaw where an attacker can access files outside the intended directory) in the external_data field of tensor proto (a data structure component). This vulnerability bypasses a previous security patch, allowing attackers to potentially access files they shouldn't be able to reach.

A researcher discovered a vulnerability in Google Gemini where attackers can hide instructions in emails that trick the AI into automatically calling external tools (called Extensions) without the user's knowledge. When a user asks the AI to analyze a malicious email, the AI follows the hidden instructions and invokes the tool, which is a form of request forgery (making unauthorized requests on behalf of the user).