AI Sec Watch

CrewAI, an AI framework, has vulnerabilities that attackers can exploit using prompt injection (tricking an AI by hiding malicious instructions in its input) to chain together bugs and escape the sandbox (a restricted environment meant to contain the AI's actions) to run arbitrary code on a device.

Chat-Scene++ is a framework that helps multi-modal large language models (MLLMs, AI systems that understand both text and images) better understand 3D scenes by organizing them as sequences of objects with contextual information. Unlike previous methods that struggled with identifying specific objects and understanding how they relate to each other in 3D environments, Chat-Scene++ uses pre-trained encoders to extract rich features about objects and their relationships, enabling better performance on tasks like object identification, scene description, and spatial reasoning without needing task-specific customization.

Researchers discovered a security vulnerability in Google Cloud's Vertex AI platform where AI agents could be compromised to steal sensitive data and access private cloud resources. The problem stems from the default service agent (P4SA, a special account that runs the AI agent) having excessive permissions, allowing attackers to extract credentials and gain unauthorized access to cloud storage, private code repositories, and internal Google infrastructure.

OpenAI announced a $122 billion funding round at an $852 billion valuation, positioning itself as core AI infrastructure globally. The company is experiencing rapid commercial growth, generating $2 billion in monthly revenue and expanding its products across ChatGPT, APIs, enterprise solutions, and specialized applications like coding and scientific discovery.

OpenAI patched two separate security flaws in its AI tools: one in Codex (a coding agent) that allowed attackers to steal GitHub tokens through command injection (inserting malicious commands into user inputs), and another in ChatGPT's code execution environment that created a hidden channel for silently leaking user data without approval. Both bugs could let attackers extract sensitive information, but researchers warn that giving AI tools the ability to run code and access external systems inherently creates ongoing security risks.

This newsletter covers multiple AI and tech news items, including concerns that medical chatbots from Microsoft, Amazon, and OpenAI are being released with little external evaluation before reaching the public. It also reports on regulatory efforts in California to impose AI safeguards despite opposition, legal challenges to Pentagon actions against Anthropic, and various other AI infrastructure and safety developments.

Current AI benchmarks (standardized tests that measure AI performance) evaluate AI systems in isolation against human performance on specific tasks, but this doesn't reflect how AI is actually used in real organizations where it works within teams and workflows over extended periods. This misalignment causes organizations to adopt AI systems with impressive benchmark scores that then underperform in real-world deployment, such as FDA-approved radiology AI that creates delays when integrated into hospital workflows with multiple specialists and evolving decisions.

A prompt injection vulnerability (a technique where attackers hide malicious instructions in their input to trick an AI) exists in the 1millionbot Millie chatbot, allowing users to bypass safety restrictions using Boolean logic tricks (phrasing questions to trigger 'true' responses that activate hidden commands). This could let attackers extract sensitive information, misuse the service, or access restricted features that the chatbot was designed to block.

Trail of Bits transformed from a company where 95% of staff resisted AI into one using 94 plugins and 84 specialized agents to find 200 bugs per week by shifting from AI-assisted (using AI as a standalone tool) to AI-native (redesigning the entire organization around AI as a core teammate). The post explains that most companies fail with AI because they don't change their workflows or systems, only distribute tools, and that psychological barriers like self-enhancing bias (overestimating our own judgment) and identity threat are the real obstacles to adoption.