AI Sec Watch

Runtime attacks on large language models are rapidly increasing, with jailbreak techniques (methods that bypass AI safety restrictions) and denial-of-service exploits (attacks that make systems unavailable) becoming more sophisticated and widely shared through open-source platforms like GitHub. The report explains that these attacks have evolved from isolated research experiments into organized toolkits accessible to threat actors, affecting production AI deployments across enterprises.

MLflow versions before 3.1.0 have a vulnerability in the gateway_proxy_handler component where it fails to properly validate the gateway_path parameter, potentially allowing SSRF (server-side request forgery, where an attacker tricks the server into making unwanted requests to internal systems). This validation gap could be exploited to access resources the attacker shouldn't be able to reach.

FastGPT, an AI Agent building platform, has a vulnerability in versions before 4.9.12 where the LastRoute parameter on the login page is not properly validated or cleaned of malicious code. This allows attackers to perform open redirect (sending users to attacker-controlled websites) or DOM-based XSS (injecting malicious JavaScript that runs in the user's browser).

Attackers can exploit large language models (LLMs) through "sponge attacks," which are denial of service (DoS) attacks that craft prompts designed to generate extremely long outputs, exhausting the model's resources and degrading performance. Researchers are developing methods to predict how long an LLM's response will be based on a given prompt, creating an early warning system to detect and prevent these resource-draining attacks.

New York's legislature passed the RAISE Act (Responsible AI Safety and Education Act), which would regulate frontier AI systems (the largest, most powerful AI models) if signed into law. The act requires developers of expensive AI models to publish safety plans, withhold unreasonably risky models from release, report safety incidents within 72 hours, and face penalties up to $10 million for violations.

The European Commission is recruiting up to 60 independent experts for a scientific panel to advise on general-purpose AI (GPAI, large AI models designed for many tasks) under the EU AI Act. The panel will assess systemic risks (widespread dangers affecting multiple countries or many users), classify AI models, and issue alerts when AI systems pose significant dangers to Europe. Applicants need a PhD in a relevant field, proven AI research experience, and independence from AI companies, with the deadline set for September 14th.

This article compares traditional application security (AppSec) practices with AI security, noting that familiar principles like input validation and authentication apply to both, but AI systems introduce unique risks. New attack types specific to AI, such as prompt injection (tricking an AI by hiding instructions in its input), model poisoning (tampering with training data), and membership inference attacks (determining if specific data was in training), require security engineers to develop new defensive strategies beyond traditional code-level vulnerability management.

Cursor, a code editor designed for AI-assisted programming, had a security flaw in versions before 0.51.0 where JSON files could automatically trigger web requests without user approval. An attacker could exploit this, especially after a prompt injection attack (tricking the AI with hidden instructions in its input), to make the AI agent send data to a malicious website.

CVE-2025-32711 is a command injection vulnerability (a weakness where an attacker tricks a program into running unintended commands) in Microsoft 365 Copilot that allows an unauthorized attacker to disclose information over a network. The vulnerability has a CVSS severity score of 4.0 (a moderate rating on a 0-10 scale where 10 is most severe). Microsoft has published information about this vulnerability, but the provided source does not contain specific technical details about the attack or its impact.