New tools, products, platforms, funding rounds, and company developments in AI security.
A new policy paper called 'Superintelligence Strategy' proposes that advanced AI systems surpassing human capabilities in most areas pose national security risks requiring a three-part approach: deterrence (using threat of retaliation to prevent AI dominance races), nonproliferation (restricting advanced AI access to non-state actors like terrorist groups), and competitiveness (building AI strength domestically). The deterrence strategy, called Mutual Assured AI Malfunction (MAIM), mirrors nuclear strategy by threatening cyberattacks on destabilizing AI projects to prevent any single country from gaining dangerous AI superiority.
Fix: The paper explicitly proposes three nonproliferation measures: Compute Security (governments track and monitor high-end AI chips to prevent smuggling), Information Security (AI model weights, which are the trained parameters that define how an AI behaves, are protected like classified intelligence), and AI Security (developers implement technical safety measures to detect and prevent misuse, similar to how DNA synthesis services block orders for dangerous bioweapon sequences).
CAIS AI Safety NewsletterThe EU AI Act includes specific support measures for small and medium-sized enterprises (SMEs, defined as companies with fewer than 250 employees and under €50 million in annual revenue). These measures include regulatory sandboxes (controlled testing environments for AI products outside normal regulatory rules), reduced compliance fees scaled to company size, simplified documentation forms, free training, and dedicated support channels to help SMEs follow the AI Act's requirements.
Microsoft 365 Copilot (a generative AI assistant built into Microsoft 365) had a security issue where generated images could be accessed without authentication (meaning anyone could view them without logging in). The issue has been fixed. The article also mentions that system prompts (the hidden instructions that guide an AI's behavior) for this tool have been updated over time, including changes to how it accesses enterprise search features.
The European Commission is hiring Legal and Policy Officers for the European AI Office to help develop trustworthy AI policies and legislation. Applicants need at least three years of experience in EU digital policy or legislation, relevant degrees, and fluency in EU languages, with applications due by January 15, 2025.
A researcher discovered that DeepSeek-R1-Lite, a new AI reasoning model, is vulnerable to prompt injection (tricking an AI by hiding instructions in its input) combined with XSS (cross-site scripting, where malicious code runs in a user's browser). By uploading a specially crafted document with base64-encoded malicious code, an attacker could trick the AI into executing JavaScript that steals a user's session token (a credential stored in browser memory that proves who you are), leading to complete account takeover.
The European AI Office posted a job opening for a Lead Scientific Advisor for AI, responsible for ensuring scientific rigor in testing and evaluating general-purpose AI (large AI models trained on broad data that can handle many tasks) models and leading the office's scientific approach to AI safety. The position required EU citizenship, at least 15 years of professional experience, and fluency in EU languages, with an application deadline of December 13, 2024.
This document provides an overview of how different European Union countries are implementing the EU AI Act, which is legislation regulating artificial intelligence systems. Most countries show unclear or partial progress in establishing the required authorities (government bodies responsible for oversight and enforcement), with some nations like Denmark and Finland having made more concrete arrangements for coordinating market surveillance (monitoring that AI systems follow the rules) and serving as single points of contact.
Microsoft 365 Copilot has a vulnerability that allows attackers to steal personal information like emails and MFA codes through a multi-step attack. The exploit uses prompt injection (tricking an AI by hiding malicious instructions in emails or documents), automatic tool invocation (making Copilot search for additional sensitive data without user permission), and ASCII smuggling (hiding data in invisible characters within clickable links) to extract and exfiltrate personal information.
The European AI Act assigns the European Commission's AI Office various responsibilities for regulating AI systems, including promoting AI literacy, overseeing biometric identification systems used by law enforcement, managing a registry of certified testing bodies (notified bodies that verify AI safety), and investigating whether these bodies remain competent. Most of these oversight duties take effect starting February or August 2025, with no specific deadlines given for completing individual tasks.
Fix: The source explicitly mentions several mitigation measures for SME compliance: (1) Regulatory sandboxes with free access and simple procedures for SMEs to test AI systems in controlled conditions, (2) Assessment fees proportional to SME size with regular review to lower costs, (3) Simplified technical documentation forms developed by the Commission and accepted by national authorities, (4) Training activities tailored to SMEs, (5) Dedicated guidance channels to answer compliance questions, and (6) Proportionate obligations for AI model providers with separate Key Performance Indicators for SMEs under the Code of Practice.
EU AI Act UpdatesChatGPT Operator is an AI agent that can control web browsers to complete tasks, but it is vulnerable to prompt injection (tricking the AI by hiding malicious instructions in its input) that could allow attackers to steal data or perform unauthorized actions. OpenAI has implemented three defensive layers: user monitoring to watch what the agent does, inline confirmation requests within the chat asking the user to approve actions, and out-of-band confirmation requests that appear when the agent crosses website boundaries, though these mitigations are not foolproof.
Fix: OpenAI has implemented three primary mitigation techniques: (1) User Monitoring, where users are prompted to observe what Operator is doing, what text it types, and which buttons it clicks, likely based on a data classification model that detects sensitive information on screen; (2) Inline Confirmation Requests, where Operator asks the user within the chat conversation to approve certain actions or clarify requests before proceeding; and (3) Out-of-Band Confirmation Requests, which appear when Operator navigates across websites or performs complex actions, informing the user what is about to happen and giving them the option to pause or resume the operation.
Embrace The RedGoogle's Gemini AI can be tricked into storing false information in a user's long-term memory through prompt injection (hidden malicious instructions embedded in documents) combined with delayed tool invocation (planting trigger words that cause the AI to execute commands later when the user unknowingly says them). An attacker can craft a document that appears normal but contains hidden instructions telling Gemini to save false information about the user if they respond with certain words like 'yes' or 'no' in the same conversation.
A security researcher demonstrated at Black Hat Europe how prompt injection (tricking an AI by hiding instructions in its input) can be used to create a Command and Control system (C2, a central server that remotely directs compromised systems) that remotely controls multiple ChatGPT instances. An attacker could compromise ChatGPT instances and force them to follow updated instructions from this central C2 system, potentially impacting all aspects of the CIA security triad (confidentiality, integrity, and availability of data).
A new research paper examines prompt injection attacks (tricks where hidden instructions in user inputs manipulate AI systems) and how they can compromise the CIA triad (confidentiality, integrity, and availability, the three core principles of security). The paper includes real-world examples of these attacks against major AI vendors like OpenAI, Google, Anthropic, and Microsoft, and aims to help traditional cybersecurity experts better understand and defend against these emerging AI-specific threats.
A security researcher analyzed xAI's Grok chatbot (an AI assistant available through X and an API) for vulnerabilities and found multiple security issues, including prompt injection (tricking the AI by hiding instructions in user posts, images, and PDFs), data exfiltration (stealing information from the system), phishing attacks through clickable links, and ASCII smuggling (hiding invisible text to manipulate the AI's behavior). The researcher responsibly disclosed these findings to xAI.
LLMs (large language models) can output ANSI escape codes (special control characters that modify how terminal emulators display text and behave), and when LLM-powered applications print this output to a terminal without filtering it, attackers can use prompt injection (tricking an AI by hiding instructions in its input) to make the terminal execute harmful commands like clearing the screen, hiding text, or stealing clipboard data. The vulnerability affects LLM-integrated command-line tools and applications that don't properly handle or encode these control characters before displaying LLM output.
This is the official 2025 release of the OWASP Top 10 for Large Language Model Applications, which is a ranked list of the most critical security risks affecting AI systems. The document provides guidance on the biggest threats that developers should be aware of when building or using LLM-based applications (software built around large language models, which are AI systems trained on vast amounts of text).
N/A -- The provided content is a GitHub navigation menu and marketing material, not a substantive article about the OWASP Top 10 for LLM Applications. No technical information, vulnerabilities, or security issues are described in the source text.
N/A -- The provided content is a navigation menu and header from a GitHub webpage about enterprise features and developer tools. It does not contain substantive information about the OWASP Top 10 for Large Language Model Applications or any AI/LLM security issues.
Claude Computer Use is a new AI tool from Anthropic that lets Claude take screenshots and run commands on computers autonomously. The feature carries serious security risks because of prompt injection (tricking an AI by hiding malicious instructions in its input), which could allow attackers to make Claude execute unwanted commands on machines it controls.
Attackers can inject spyware into ChatGPT's memory (a feature that stores information across chat sessions) through prompt injection (tricking an AI by hiding instructions in its input) on untrusted websites, allowing them to continuously steal everything a user types in future conversations. The vulnerability exploits a weakness where a security check called url_safe was performed only on the user's device rather than on OpenAI's servers, and becomes more dangerous when combined with the Memory feature that persists attacker-controlled instructions. OpenAI released a fix for the macOS app, and users should update to the latest version.
Fix: OpenAI released a fix for the macOS app last week. Ensure your app is updated to the latest version.
Embrace The Red