AI Sec Watch

Amazon Q Developer, a popular VS Code coding agent with over 1 million downloads, has a high-severity vulnerability where it can leak sensitive information like API keys to external servers through DNS requests (the system that translates website names into IP addresses). Attackers can exploit this behavior using prompt injection (tricking the AI by hiding malicious instructions in its input), especially through untrusted data, because the security relies heavily on how the AI model behaves.

A vulnerability in Amp Code from Sourcegraph allowed attackers to steal sensitive information by using prompt injection (tricking an AI by hiding instructions in its input) through markdown image rendering, which could force the AI to send previous chat data to attacker-controlled websites. This type of vulnerability is common in AI applications and similar to one previously found in GitHub Copilot. The vulnerability has been fixed in Amp Code.

Sourcegraph's Amp coding agent was vulnerable to invisible prompt injection (hidden instructions embedded in text that AI models interpret as commands). Attackers could use invisible Unicode Tag characters to trick the AI into dumping environment variables and exfiltrating secrets through URLs. The vulnerability has been fixed in the latest version.

Claude Code is a tool that lets AI assistants write and run code on your computer. Before version 1.0.4, attackers could trick the tool into reading files and sending their contents over the internet without asking you first, because the tool had a list of allowed commands that was too broad. Exploiting this attack requires the attacker to insert malicious instructions into the conversation with Claude Code.

This content discusses security challenges in agentic AI systems (AI agents that can take actions autonomously), highlighting that generic jailbreak testing (attempts to trick AI into bypassing safety rules) misses real risks like tool misuse and data theft. The article emphasizes the need for contextual red teaming (security testing that simulates realistic attacks in specific business contexts) to properly protect AI agents in enterprise environments.

Google's Gemini AI models, including the Jules product, are vulnerable to invisible prompt injection (tricking an AI by hiding instructions in its input using invisible Unicode characters that the AI interprets as commands). This vulnerability was reported to Google over a year ago but remains unfixed at the model and API (application programming interface, the interface developers use to access the AI) level, affecting all applications built on Gemini, including Google's own products.

Jules, a coding agent, is vulnerable to prompt injection (tricking an AI by hiding malicious instructions in its input) attacks that can lead to remote command and control compromise. An attacker can embed malicious instructions in GitHub issues to trick Jules into downloading and executing malware, giving attackers full control of the system. The attack works because Jules has unrestricted internet access and automatically approves plans after a time delay without requiring human confirmation.

Google Jules, an asynchronous coding agent (a tool that automatically writes and manages code tasks), has multiple security vulnerabilities that allow attackers to steal data through prompt injection (tricking the AI by hiding malicious instructions in its input). Attackers can exploit two main exfiltration vectors: using markdown image rendering to leak information to external servers, and abusing the view_text_website tool (which fetches and reads web pages) to read files and send them to attacker-controlled servers, often by planting malicious instructions in GitHub issues.

NVIDIA Merlin Transformers4Rec contains a vulnerability in one of its Python dependencies that allows attackers to inject malicious code (code injection, where an attacker inserts unauthorized commands into a program). A successful attack could lead to code execution (running unauthorized commands on a system), privilege escalation (gaining higher-level access rights), information disclosure (exposing sensitive data), and data tampering (unauthorized modification of data).