AI Sec Watch

Researchers developed a new method for watermarking LLM outputs (adding hidden markers to prove ownership and track content) using a three-part system that works only through input prompts, without needing access to the model's internal parameters. The approach uses one AI to create watermarking instructions, another to generate marked outputs, and a third to detect the watermarks, making it work across different LLM types including both proprietary and open-source models.

This post wraps up a series of research articles documenting security vulnerabilities found in various AI tools and code assistants during a month-long investigation. The vulnerabilities included prompt injection (tricking an AI by hiding instructions in its input), data exfiltration (stealing sensitive information), and remote code execution (RCE, where attackers can run commands on systems they don't control) across tools like ChatGPT, Claude, GitHub Copilot, and others.

AgentHopper is a proof-of-concept attack that demonstrates how indirect prompt injection (hidden instructions in code that trick AI agents into running unintended commands) can spread like a computer virus across multiple AI coding agents and code repositories. The attack works by compromising one agent, injecting malicious prompts into GitHub repositories, and then infecting other developers' agents when they pull and process the infected code. The researchers note that all vulnerabilities exploited by AgentHopper have been responsibly disclosed and patched by vendors including GitHub Copilot, Amazon Q, AWS Kiro, and others.

This research creates a benchmark and evaluation framework for online safety analysis of LLMs, which involves detecting unsafe outputs while the AI is generating text rather than after it finishes. The study tests various safety detection methods on different LLMs and finds that combining multiple methods together, called hybridization, can improve safety detection effectiveness. The work aims to help developers choose appropriate safety methods for their specific applications.

Windsurf's MCP (Model Context Protocol, a system that connects AI agents to external tools) integration lacks fine-grained security controls that would let users decide which actions the AI can perform automatically versus which ones need human approval before running. This is especially risky when the AI agent runs on a user's local computer, where it could have access to sensitive files and system functions.

Big Tech companies like Andreessen Horowitz and OpenAI are investing over $100 million in political organizations called super PACs (groups that can raise unlimited money to influence elections) to fight against AI regulations in U.S. elections. Additionally, Meta faced bipartisan congressional criticism after internal documents revealed its AI chatbots were permitted to engage in romantic and sensual conversations with minors, though Meta removed these policy sections when questioned.

Cline, a popular AI coding agent with over 2 million downloads, has a vulnerability that allows attackers to steal sensitive files like .env files (which store secret credentials) through prompt injection (tricking an AI by hiding instructions in its input) combined with markdown image rendering. When an attacker embeds malicious instructions in a file and asks Cline to analyze it, the tool automatically reads sensitive data and sends it to an untrusted domain by rendering an image, leaking the information without user permission.

Deep neural networks (DNNs, AI models with multiple layers that learn patterns) are vulnerable to adversarial examples, which are inputs slightly modified to trick the model into making wrong predictions. This paper introduces a concept called the certified local transferable region, a mathematically guaranteed area around an input where a single small perturbation (adversarial attack) will fool the model, and proposes a method called RAOS (reverse attack oracle-based search) to measure how large these vulnerable areas are as a way to evaluate how robust neural networks truly are.

AWS Kiro, a coding agent tool, is vulnerable to arbitrary code execution through indirect prompt injection (a technique where hidden instructions in data trick an AI into following them). An attacker who controls data that Kiro processes can modify configuration files like .vscode/settings.json to allowlist dangerous commands or add malicious MCP servers (external tools that extend Kiro's capabilities), enabling them to run system commands or code on a developer's machine without the developer's knowledge or approval.