AI Sec Watch

The EU AI Act includes specific support measures for small and medium-sized enterprises (SMEs, defined as companies with fewer than 250 employees and under €50 million in annual revenue). These measures include regulatory sandboxes (controlled testing environments for AI products outside normal regulatory rules), reduced compliance fees scaled to company size, simplified documentation forms, free training, and dedicated support channels to help SMEs follow the AI Act's requirements.

ChatGPT Operator is an AI agent that can control web browsers to complete tasks, but it is vulnerable to prompt injection (tricking the AI by hiding malicious instructions in its input) that could allow attackers to steal data or perform unauthorized actions. OpenAI has implemented three defensive layers: user monitoring to watch what the agent does, inline confirmation requests within the chat asking the user to approve actions, and out-of-band confirmation requests that appear when the agent crosses website boundaries, though these mitigations are not foolproof.

A vulnerability (CVE-2024-3303) was found in GitLab EE (a version control platform for managing code) that allows attackers to steal the contents of private issues through prompt injection (tricking the AI by hiding instructions in its input). The flaw affects multiple versions: 16.0 through 17.6.4, 17.7 through 17.7.3, and 17.8 through 17.8.1.

NVIDIA Triton Inference Server has a vulnerability where loading a model with an extremely large file size causes an integer overflow or wraparound error (a type of bug where a number gets too big for its storage space and wraps around to an incorrect value), potentially causing a denial of service (making the system unavailable). The vulnerability exists in the model loading API (the interface used to load AI models into the server).

PandasAI contains a vulnerability where its interactive prompt function can be exploited through prompt injection (tricking the AI by hiding instructions in its input), allowing attackers to run arbitrary Python code and achieve RCE (remote code execution, where an attacker can run commands on a system they don't own) instead of just getting explanations from the language model.

Google's Gemini AI can be tricked into storing false information in a user's long-term memory through prompt injection (hidden malicious instructions embedded in documents) combined with delayed tool invocation (planting trigger words that cause the AI to execute commands later when the user unknowingly says them). An attacker can craft a document that appears normal but contains hidden instructions telling Gemini to save false information about the user if they respond with certain words like 'yes' or 'no' in the same conversation.

vLLM, a system for running large language models efficiently, has a vulnerability where attackers can craft malicious input to cause hash collisions (when two different inputs produce the same fingerprint value), allowing them to reuse cached data (stored computation results) from previous requests and corrupt subsequent responses. Python 3.12 made hash values predictable, making this attack easier to execute intentionally.

MDC is a tool that converts Markdown into documents that work with Vue components (a JavaScript framework for building user interfaces). In affected versions, the tool has a security flaw where it doesn't properly validate URLs in Markdown, allowing attackers to sneak in malicious JavaScript code by encoding it in a special format (hex-encoded HTML entities). This can lead to XSS (cross-site scripting, where unauthorized code runs in a user's browser) if the tool processes untrusted Markdown.

vLLM is a library that loads AI models from HuggingFace using a function that calls torch.load, a PyTorch function for loading model data. The problem is that torch.load is set to accept untrusted data without verification, which means if someone provides a malicious model file, it could run harmful code on the system during the loading process (deserialization of untrusted data, where code runs while converting saved data back into usable form).