AI Sec Watch

This article argues that there are fundamental mathematical limits to how secure and well-aligned (following intended behavior) AI systems can be, and that understanding these limits is important before deploying AI widely. The research also shows that AI systems have basic reasoning limitations that stem from these same information-theoretic constraints.

This paper addresses transferable targeted attacks (TTAs), which are adversarial techniques that trick AI models by creating modified images that fool multiple different models, not just one. The researchers found that simple scaling transformations (resizing images at different sizes) are surprisingly effective for these attacks and proposed S4ST, a method that combines scaling with other transformations to achieve better results without needing access to the victim model's training data.

DreamFuse is a new AI system designed to seamlessly blend foreground objects (like people or animals) into background scenes to create realistic composite images. The system uses a diffusion model (a type of AI that generates images by gradually refining random noise) with a Diffusion Transformer architecture and special attention mechanisms to understand how foreground and background elements should interact, supporting tasks like object insertion, replacement, and editing based on text descriptions.

MLflow has a security flaw called an authorization bypass (a weakness where access controls are not properly checked) in its AJAX endpoint (a web interface used to download model files) that allows users without permission to download saved model artifacts they shouldn't be able to access. This affects MLflow versions up to 3.10.1 and has a CVSS score (a 0-10 rating of severity) of 5.3, considered medium severity.

MLflow has a stored XSS vulnerability (cross-site scripting, where malicious code hidden in data executes when viewed in a web browser) in how it handles YAML-based MLmodel artifact files. An authenticated attacker can upload a specially crafted MLmodel file that runs malicious code when another user views it in the web interface, potentially letting the attacker hijack sessions or perform actions as that user. This affects MLflow version 3.10.1 and earlier.

GrafanaGhost is a critical vulnerability in Grafana (a data visualization platform) that uses indirect prompt injection (tricking an AI by hiding malicious instructions in data it processes) to steal sensitive enterprise data without requiring user authentication or interaction. Attackers chain together multiple exploits, including bypassing URL validation and AI safety guardrails, to trick Grafana's AI into sending confidential information to attacker-controlled servers.

Attackers are targeting over 1,000 publicly accessible ComfyUI instances (a platform for running AI image generation) with an automated scanner that exploits a misconfiguration allowing unauthenticated remote code execution (the ability to run commands on a system without permission). Once compromised, these systems are enrolled in botnets (networks of infected computers controlled remotely) to mine cryptocurrency and serve as proxies.

OpenAI has published policy proposals suggesting that companies should trial four-day work weeks as AI tools become more capable and potentially displace workers from jobs. The company argues that AI systems will soon complete projects in days that currently take months, and recommends employers offer benefits like reduced work hours without pay cuts, increased retirement contributions, and subsidized childcare to help workers adapt to this shift.

Broadcom, a chip designer, announced new deals to produce AI chips for Google and expanded its partnership with Anthropic (an AI company), causing its stock price to rise 3.7% in premarket trading. The deals include revenue commitments and access to computing capacity, which analysts believe signal strong future demand for custom AI chips and may ease investor concerns about competition.