AI Sec Watch

Omio, a travel platform connecting millions of travelers with transportation options, is using conversational AI (AI that understands natural language questions from users) to let people book trips by simply describing where they want to go, rather than searching through websites. The company launched this capability through ChatGPT in 2023 by connecting OpenAI's language models directly to its real-time transportation data, and it is now using similar AI tools internally to help engineers and other employees work more efficiently.

Researchers discovered that AI models struggle to distinguish between their own internal instructions (wrapped in tags like <system> and <think>) and untrusted user input (wrapped in <user> tags), a problem called role confusion. The models pay more attention to the writing style of text than its actual meaning, allowing attackers to craft jailbreaks (unauthorized bypasses of safety rules) by mimicking the style of internal thinking blocks. However, rewriting malicious text in a different style (called 'destyling') significantly reduced attack success rates from 61% to 10%, showing that format changes can help models better distinguish between trusted and untrusted content.

This article describes successfully porting the Moebius image inpainting model (a small AI model that can remove objects from images and fill in the missing areas) to run in a web browser using WebGPU (a graphics technology that lets browsers use GPU acceleration). The author used Claude Code, an AI coding agent, to help convert the model from Python and NVIDIA CUDA (specialized GPU software for training AI) into a web-compatible format using ONNX Runtime Web.

Budibase has a DNS rebinding vulnerability (a type of attack where DNS lookups return different IP addresses at different times) in its SSRF protection. The software checks if a hostname is safe by looking up its IP address and checking a blacklist, but then performs a separate DNS lookup when actually connecting. An attacker controlling DNS can return a public IP during the safety check and a private/internal IP during the actual connection, allowing them to access internal services like localhost or cloud metadata endpoints.

LangChain, a framework for building AI agents and applications powered by large language models, had a vulnerability before version 1.3.9 where several components that work with file paths did not properly restrict access to files. This meant attackers could use glob patterns (wildcards for matching multiple files), symlinks (shortcuts to files), or specially crafted paths to read files outside the intended directory, especially when an AI system processes untrusted input. The vulnerability allowed unauthorized file disclosure.

Anthropic updated its privacy policy to allow Claude users to appeal account flags by uploading government-issued ID documents and biometric data (facial scans and face geometry templates, which are digital measurements of facial features). The policy applies only to a small subset of users whose accounts are flagged for fraud rather than immediately banned, and Anthropic says it uses this verification to comply with various legal requirements and security measures.

The US government placed export controls on Anthropic's AI model Fable, claiming it posed a national security threat because it was very good at writing code. The incident has sparked concerns about unintended consequences: companies may switch to cheaper Chinese open-source AI models with fewer safety guardrails, the cybersecurity community fears losing access to Anthropic's models for defensive research, and lawmakers may introduce new regulations around AI development and military use.

Microsoft fixed a vulnerability chain called AutoJack in AutoGen Studio, a graphical tool for building multi-agent AI systems (where multiple AI programs work together). The flaw let attackers trick an AI agent into running arbitrary commands (unrestricted code) on the host system just by having a developer visit a malicious webpage. The vulnerability was caught before any official release, so only developers building directly from GitHub source code during a brief window were affected.

Check Point, a security company serving over 100,000 customers, has partnered with OpenAI to integrate advanced AI models into its cybersecurity products through OpenAI's Daybreak Cyber Partner Program. This integration aims to improve threat prevention, speed up incident response (remediation, the process of fixing security issues), and strengthen security operations for their customers.