AI Sec Watch

ArXiv, a platform where researchers share academic papers before formal publication, is implementing new rules to reduce AI slop (low-quality or unreliable AI-generated content). Authors who submit papers with clear evidence they didn't check their AI outputs, such as hallucinated references (false citations made up by the AI) or leftover comments from an LLM (large language model, an AI trained on massive amounts of text), will be banned from the platform for one year and must have future papers accepted at a reputable peer-reviewed venue.

Open WebUI, a self-hosted AI platform that runs offline, had a vulnerability before version 0.9.0 where certain API endpoints (like /api/generate and /api/embeddings) accepted any model name from users and sent requests to the backend without checking if those users had permission to use that model. The endpoints only verified that a user was logged in and that the model existed, but skipped the access control check (AccessGrants.has_access(), which determines what resources a user is allowed to access).

Open WebUI, a self-hosted AI platform that runs offline, had a security flaw in versions before 0.9.0 where the /responses endpoint allowed any logged-in user to access any model on the system without proper permission checks. While the main chat endpoint verified that users had the right to use specific models through ownership, group membership, and access grants, the /responses proxy skipped these checks and only confirmed the user was logged in, letting attackers use models they shouldn't have access to.

OpenAI reorganized its leadership structure to focus on developing AI agents (AI systems that can independently perform multiple tasks and make decisions). The company plans to merge ChatGPT and Codex (a code-generation AI tool) into a single unified platform designed around this agentic approach, with president Greg Brockman now overseeing all product decisions.

Budibase's AI Extract File automation step has a server-side request forgery vulnerability (SSRF, a type of attack where a server makes requests to internal addresses it shouldn't access) because it uses `fetch()` directly without IP blacklist validation. Every other automation step in the same codebase properly uses `fetchWithBlacklist()` to block requests to internal networks like 127.0.0.1 and 169.254.169.254, but the AI step bypasses these protections, allowing authenticated users to access cloud metadata, scan internal networks, and potentially steal credentials.

The AWS AI Security Framework is a structured approach that helps organizations secure AI systems by applying the right security controls across three layers (infrastructure, identity/data, and AI application), three use cases (question-answering AI, data-connected AI like RAG, and autonomous agents), and three phases (prototype, production, and scale). The framework addresses unique AI security challenges like prompt injection (tricking AI systems by hiding malicious instructions in user input) and non-deterministic outputs by implementing input validation, content filtering, and continuous monitoring from day one of development.

Microsoft APM is a tool that manages dependencies for AI agents, and versions before 0.13.0 have a security flaw on Windows systems. When installing a bundle (a package of code) from a .tar.gz file (a compressed archive format), the tool extracts files without properly checking if any file paths could escape the intended folder, potentially allowing an attacker to place files anywhere on the system by using absolute paths like D:/.

Microsoft APM, a dependency manager for AI agents, had a vulnerability in versions 0.5.4 to 0.12.4 where symbolic links (shortcuts that point to other files) in downloaded packages were followed without checking, potentially allowing attackers to read or write arbitrary files on a developer's machine. The vulnerability went undetected by security checks because the resulting files were not flagged by the package hash verification, security scans, or audit tools.

Microsoft APM is a tool that manages dependencies (external code libraries) for AI agents. Before version 0.8.12, it had a path traversal vulnerability (a security flaw where an attacker can access files outside the intended directory) that allowed malicious plugins to copy arbitrary files from a user's computer during installation by using absolute paths or '../' sequences to escape the plugin directory.