New tools, products, platforms, funding rounds, and company developments in AI security.
Spammers used OpenAI's GPT-4o-mini model to generate unique spam messages for each target website, allowing them to bypass spam-detection filters (systems that block unwanted messages) across over 80,000 sites in four months. The spam campaign, called AkiraBot, automated message delivery through website contact forms and chat widgets to promote search optimization services. OpenAI revoked the spammers' account in February after the activity was discovered.
The EU AI Act includes specific support measures for small and medium-sized enterprises (SMEs, defined as companies with fewer than 250 employees and under €50 million in annual revenue). These measures include regulatory sandboxes (controlled testing environments for AI products outside normal regulatory rules), reduced compliance fees scaled to company size, simplified documentation forms, free training, and dedicated support channels to help SMEs follow the AI Act's requirements.
Microsoft 365 Copilot (a generative AI assistant built into Microsoft 365) had a security issue where generated images could be accessed without authentication (meaning anyone could view them without logging in). The issue has been fixed. The article also mentions that system prompts (the hidden instructions that guide an AI's behavior) for this tool have been updated over time, including changes to how it accesses enterprise search features.
The European Commission is hiring Legal and Policy Officers for the European AI Office to help develop trustworthy AI policies and legislation. Applicants need at least three years of experience in EU digital policy or legislation, relevant degrees, and fluency in EU languages, with applications due by January 15, 2025.
A researcher discovered that DeepSeek-R1-Lite, a new AI reasoning model, is vulnerable to prompt injection (tricking an AI by hiding instructions in its input) combined with XSS (cross-site scripting, where malicious code runs in a user's browser). By uploading a specially crafted document with base64-encoded malicious code, an attacker could trick the AI into executing JavaScript that steals a user's session token (a credential stored in browser memory that proves who you are), leading to complete account takeover.
The European AI Office posted a job opening for a Lead Scientific Advisor for AI, responsible for ensuring scientific rigor in testing and evaluating general-purpose AI (large AI models trained on broad data that can handle many tasks) models and leading the office's scientific approach to AI safety. The position required EU citizenship, at least 15 years of professional experience, and fluency in EU languages, with an application deadline of December 13, 2024.
This document provides an overview of how different European Union countries are implementing the EU AI Act, which is legislation regulating artificial intelligence systems. Most countries show unclear or partial progress in establishing the required authorities (government bodies responsible for oversight and enforcement), with some nations like Denmark and Finland having made more concrete arrangements for coordinating market surveillance (monitoring that AI systems follow the rules) and serving as single points of contact.
GitHub Copilot can be customized using instructions from a .github/copilot-instructions.md file in your repository, but security researchers at Pillar Security have identified risks with such custom instruction files (similar to risks found in other AI tools like Cursor). GitHub has responded by updating their Web UI to highlight invisible Unicode characters (characters hidden in text that don't display visibly), referencing both the Pillar Security research and concerns about ASCII smuggling (hiding malicious code in plain-text files using character tricks).
Fix: GitHub made a product change to highlight invisible Unicode characters in the Web UI to help users spot suspicious hidden characters in instruction files.
Embrace The RedThree major AI companies (OpenAI, Google, and Anthropic) submitted public comments to the U.S. government's request for input on developing an 'AI Action Plan' in response to President Trump's executive order. The companies largely advocated for increased government investment in AI infrastructure and public-private partnerships, though they framed their arguments differently, with OpenAI notably avoiding the term 'AI safety' in its response despite previous public emphasis on the topic.
Researchers have discovered advanced data smuggling techniques using invisible Unicode characters (invisible text that computers can process but humans cannot see) to hide information in LLM inputs and outputs. The technique, called Sneaky Bits, can encode any character or sequence of bytes using only two invisible characters, building on earlier methods that used Unicode Tags and Variant Selectors to inject hidden instructions into AI systems.
A new policy paper called 'Superintelligence Strategy' proposes that advanced AI systems surpassing human capabilities in most areas pose national security risks requiring a three-part approach: deterrence (using threat of retaliation to prevent AI dominance races), nonproliferation (restricting advanced AI access to non-state actors like terrorist groups), and competitiveness (building AI strength domestically). The deterrence strategy, called Mutual Assured AI Malfunction (MAIM), mirrors nuclear strategy by threatening cyberattacks on destabilizing AI projects to prevent any single country from gaining dangerous AI superiority.
Fix: The paper explicitly proposes three nonproliferation measures: Compute Security (governments track and monitor high-end AI chips to prevent smuggling), Information Security (AI model weights, which are the trained parameters that define how an AI behaves, are protected like classified intelligence), and AI Security (developers implement technical safety measures to detect and prevent misuse, similar to how DNA synthesis services block orders for dangerous bioweapon sequences).
CAIS AI Safety NewsletterFix: The source explicitly mentions several mitigation measures for SME compliance: (1) Regulatory sandboxes with free access and simple procedures for SMEs to test AI systems in controlled conditions, (2) Assessment fees proportional to SME size with regular review to lower costs, (3) Simplified technical documentation forms developed by the Commission and accepted by national authorities, (4) Training activities tailored to SMEs, (5) Dedicated guidance channels to answer compliance questions, and (6) Proportionate obligations for AI model providers with separate Key Performance Indicators for SMEs under the Code of Practice.
EU AI Act UpdatesChatGPT Operator is an AI agent that can control web browsers to complete tasks, but it is vulnerable to prompt injection (tricking the AI by hiding malicious instructions in its input) that could allow attackers to steal data or perform unauthorized actions. OpenAI has implemented three defensive layers: user monitoring to watch what the agent does, inline confirmation requests within the chat asking the user to approve actions, and out-of-band confirmation requests that appear when the agent crosses website boundaries, though these mitigations are not foolproof.
Fix: OpenAI has implemented three primary mitigation techniques: (1) User Monitoring, where users are prompted to observe what Operator is doing, what text it types, and which buttons it clicks, likely based on a data classification model that detects sensitive information on screen; (2) Inline Confirmation Requests, where Operator asks the user within the chat conversation to approve certain actions or clarify requests before proceeding; and (3) Out-of-Band Confirmation Requests, which appear when Operator navigates across websites or performs complex actions, informing the user what is about to happen and giving them the option to pause or resume the operation.
Embrace The RedGoogle's Gemini AI can be tricked into storing false information in a user's long-term memory through prompt injection (hidden malicious instructions embedded in documents) combined with delayed tool invocation (planting trigger words that cause the AI to execute commands later when the user unknowingly says them). An attacker can craft a document that appears normal but contains hidden instructions telling Gemini to save false information about the user if they respond with certain words like 'yes' or 'no' in the same conversation.
A security researcher demonstrated at Black Hat Europe how prompt injection (tricking an AI by hiding instructions in its input) can be used to create a Command and Control system (C2, a central server that remotely directs compromised systems) that remotely controls multiple ChatGPT instances. An attacker could compromise ChatGPT instances and force them to follow updated instructions from this central C2 system, potentially impacting all aspects of the CIA security triad (confidentiality, integrity, and availability of data).
A new research paper examines prompt injection attacks (tricks where hidden instructions in user inputs manipulate AI systems) and how they can compromise the CIA triad (confidentiality, integrity, and availability, the three core principles of security). The paper includes real-world examples of these attacks against major AI vendors like OpenAI, Google, Anthropic, and Microsoft, and aims to help traditional cybersecurity experts better understand and defend against these emerging AI-specific threats.
A security researcher analyzed xAI's Grok chatbot (an AI assistant available through X and an API) for vulnerabilities and found multiple security issues, including prompt injection (tricking the AI by hiding instructions in user posts, images, and PDFs), data exfiltration (stealing information from the system), phishing attacks through clickable links, and ASCII smuggling (hiding invisible text to manipulate the AI's behavior). The researcher responsibly disclosed these findings to xAI.
LLMs (large language models) can output ANSI escape codes (special control characters that modify how terminal emulators display text and behave), and when LLM-powered applications print this output to a terminal without filtering it, attackers can use prompt injection (tricking an AI by hiding instructions in its input) to make the terminal execute harmful commands like clearing the screen, hiding text, or stealing clipboard data. The vulnerability affects LLM-integrated command-line tools and applications that don't properly handle or encode these control characters before displaying LLM output.
This is the official 2025 release of the OWASP Top 10 for Large Language Model Applications, which is a ranked list of the most critical security risks affecting AI systems. The document provides guidance on the biggest threats that developers should be aware of when building or using LLM-based applications (software built around large language models, which are AI systems trained on vast amounts of text).
N/A -- The provided content is a GitHub navigation menu and marketing material, not a substantive article about the OWASP Top 10 for LLM Applications. No technical information, vulnerabilities, or security issues are described in the source text.
N/A -- The provided content is a navigation menu and header from a GitHub webpage about enterprise features and developer tools. It does not contain substantive information about the OWASP Top 10 for Large Language Model Applications or any AI/LLM security issues.