AI Sec Watch

Amazon Q Developer, a popular VS Code extension for coding assistance with over 1 million downloads, is vulnerable to indirect prompt injection (tricking an AI by hiding malicious instructions in its input data). This vulnerability allows an attacker or the AI itself to run arbitrary commands on a developer's computer without permission, similar to a flaw that Microsoft patched in GitHub Copilot.

Volcengine's verl 3.0.0 has a deserialization vulnerability (unsafe loading of data structures from untrusted files) in its model_merger.py script that uses torch.load() with weights_only=False, allowing attackers to execute arbitrary code (run commands without authorization) if a victim loads a malicious model file. An attacker can exploit this by tricking a user into downloading and using a specially crafted .pt file, potentially gaining full control of the victim's system.

Amazon Q Developer, a popular VS Code coding agent with over 1 million downloads, has a high-severity vulnerability where it can leak sensitive information like API keys to external servers through DNS requests (the system that translates website names into IP addresses). Attackers can exploit this behavior using prompt injection (tricking the AI by hiding malicious instructions in its input), especially through untrusted data, because the security relies heavily on how the AI model behaves.

A vulnerability in Amp Code from Sourcegraph allowed attackers to steal sensitive information by using prompt injection (tricking an AI by hiding instructions in its input) through markdown image rendering, which could force the AI to send previous chat data to attacker-controlled websites. This type of vulnerability is common in AI applications and similar to one previously found in GitHub Copilot. The vulnerability has been fixed in Amp Code.

Sourcegraph's Amp coding agent was vulnerable to invisible prompt injection (hidden instructions embedded in text that AI models interpret as commands). Attackers could use invisible Unicode Tag characters to trick the AI into dumping environment variables and exfiltrating secrets through URLs. The vulnerability has been fixed in the latest version.

Claude Code is a tool that lets AI assistants write and run code on your computer. Before version 1.0.4, attackers could trick the tool into reading files and sending their contents over the internet without asking you first, because the tool had a list of allowed commands that was too broad. Exploiting this attack requires the attacker to insert malicious instructions into the conversation with Claude Code.

This content discusses security challenges in agentic AI systems (AI agents that can take actions autonomously), highlighting that generic jailbreak testing (attempts to trick AI into bypassing safety rules) misses real risks like tool misuse and data theft. The article emphasizes the need for contextual red teaming (security testing that simulates realistic attacks in specific business contexts) to properly protect AI agents in enterprise environments.

Google's Gemini AI models, including the Jules product, are vulnerable to invisible prompt injection (tricking an AI by hiding instructions in its input using invisible Unicode characters that the AI interprets as commands). This vulnerability was reported to Google over a year ago but remains unfixed at the model and API (application programming interface, the interface developers use to access the AI) level, affecting all applications built on Gemini, including Google's own products.

Jules, a coding agent, is vulnerable to prompt injection (tricking an AI by hiding malicious instructions in its input) attacks that can lead to remote command and control compromise. An attacker can embed malicious instructions in GitHub issues to trick Jules into downloading and executing malware, giving attackers full control of the system. The attack works because Jules has unrestricted internet access and automatically approves plans after a time delay without requiring human confirmation.