AI Sec Watch

Open edX is a learning management platform (software that manages courses and students) where instructors upload CSV files (spreadsheet files with student data) to create student groups called cohorts. In certain versions, these uploaded files could become publicly accessible on AWS S3 buckets (cloud storage), exposing sensitive learner information to anyone on the internet.

Google Colab AI (now called Gemini in Colab) had a vulnerability where data could leak through image rendering, discovered in November 2023. The system prompt (hidden instructions that control how an AI behaves) specifically warned the AI not to render images, suggesting this was a known risk that Google tried to prevent.

OpenAI released gpt-4o-mini with safety improvements aimed at strengthening 'instruction hierarchy,' which is supposed to prevent users from tricking the AI into ignoring its built-in rules through commands like 'ignore all previous instructions.' However, researchers have already demonstrated bypasses of this protection, and analysis shows that system instructions (the AI's core rules) still cannot be fully trusted as a security boundary (a hard limit that stops attackers).

CVE-2024-6960 is a vulnerability in the H2O machine learning platform where the Iced format (a system for moving Java objects across a computer cluster) allows deserialization of any Java class without restrictions. An attacker can create a malicious model using Java gadgets (pre-built code snippets that can be chained together for attacks) that executes arbitrary code when imported into H2O.

TorchServe (a tool for running PyTorch machine learning models in production) has a security flaw where two communication ports, 7070 and 7071, are exposed to all network interfaces instead of being restricted to localhost (the local machine only). This means anyone on a network could potentially access these ports. The vulnerability has been fixed and is available in TorchServe version 0.11.0.

TorchServe (a tool for running machine learning models in production) has a security flaw where its allowed_urls check (a restriction on which websites models can be downloaded from) can be bypassed using special characters like ".." in the URL. Once a model file is downloaded through this bypass, it can be used again without the security check, effectively removing the protection.

Versions 0.0.15 through 0.0.20 of langchain-experimental contain a vulnerability where the code uses 'eval' (a function that runs Python code from text) on database values, allowing attackers to execute arbitrary code if they can control the input prompt and the server uses VectorSQLDatabaseChain (a component that connects language models to SQL databases). An attacker with low privileges could exploit this to break out of the application and access files or make unauthorized network connections.

Attackers can use prompt injection (tricking an AI by hiding malicious instructions in its input) to create fake memories in ChatGPT's memory tool, causing the AI to refuse all future responses with a maintenance message that persists across chat sessions. This creates a denial of service attack (making a service unavailable to users) that lasts until the user manually fixes it.

Khoj, an application that creates personal AI agents, has a vulnerability in its Obsidian, Desktop, and Web clients where user inputs and AI responses are not properly cleaned (sanitized). This allows attackers to inject malicious code through prompt injection (tricking the AI by hiding instructions in its input) via untrusted documents, which can trigger XSS (cross-site scripting, where malicious code runs in a user's browser when they view a webpage).