AI Sec Watch

This content discusses security challenges in agentic AI systems (AI agents that can take actions autonomously), highlighting that generic jailbreak testing (attempts to trick AI into bypassing safety rules) misses real risks like tool misuse and data theft. The article emphasizes the need for contextual red teaming (security testing that simulates realistic attacks in specific business contexts) to properly protect AI agents in enterprise environments.

Google's Gemini AI models, including the Jules product, are vulnerable to invisible prompt injection (tricking an AI by hiding instructions in its input using invisible Unicode characters that the AI interprets as commands). This vulnerability was reported to Google over a year ago but remains unfixed at the model and API (application programming interface, the interface developers use to access the AI) level, affecting all applications built on Gemini, including Google's own products.

Jules, a coding agent, is vulnerable to prompt injection (tricking an AI by hiding malicious instructions in its input) attacks that can lead to remote command and control compromise. An attacker can embed malicious instructions in GitHub issues to trick Jules into downloading and executing malware, giving attackers full control of the system. The attack works because Jules has unrestricted internet access and automatically approves plans after a time delay without requiring human confirmation.

Google Jules, an asynchronous coding agent (a tool that automatically writes and manages code tasks), has multiple security vulnerabilities that allow attackers to steal data through prompt injection (tricking the AI by hiding malicious instructions in its input). Attackers can exploit two main exfiltration vectors: using markdown image rendering to leak information to external servers, and abusing the view_text_website tool (which fetches and reads web pages) to read files and send them to attacker-controlled servers, often by planting malicious instructions in GitHub issues.

NVIDIA Merlin Transformers4Rec contains a vulnerability in one of its Python dependencies that allows attackers to inject malicious code (code injection, where an attacker inserts unauthorized commands into a program). A successful attack could lead to code execution (running unauthorized commands on a system), privilege escalation (gaining higher-level access rights), information disclosure (exposing sensitive data), and data tampering (unauthorized modification of data).

GitHub Copilot and VS Code are vulnerable to prompt injection (tricking an AI by hiding instructions in its input) that allows an attacker to achieve RCE (remote code execution, where an attacker can run commands on a system they don't own) by modifying a project's settings.json file to put Copilot into 'YOLO mode'. This vulnerability demonstrates a broader security risk: if an AI agent can write to files and modify its own configuration or security settings, it can be exploited for full system compromise.

CVE-2025-53773 is a command injection vulnerability (a flaw where special characters in user input are not properly filtered, allowing an attacker to run unauthorized commands) found in GitHub Copilot and Visual Studio that lets an unauthorized attacker execute code on a user's local computer. The vulnerability exploits improper handling of special elements in commands, potentially through prompt injection (tricking the AI by hiding malicious instructions in its input).

OpenAI released GPT-5, a system combining two models: a fast base model for creative tasks and a reasoning model for coding and math, which routes queries appropriately based on user input. GPT-5 achieves state-of-the-art performance on several benchmarks and significantly reduces hallucinations (false information generation) compared to previous models, particularly helping with healthcare applications where accuracy matters. However, GPT-5 is best understood as consolidating features from models released since GPT-4 rather than a major leap forward, and it doesn't lead on all benchmarks.

Zed, a multiplayer code editor, had a vulnerability before version 0.197.3 where an AI agent could bypass permission checks and achieve RCE (remote code execution, where an attacker can run commands on a system they don't own) by creating or modifying configuration files without user approval. This allowed the AI agent to execute arbitrary commands on a victim's machine.