New tools, products, platforms, funding rounds, and company developments in AI security.
The EU AI Act requires member states to receive and register notifications about high-risk AI systems (AI systems that pose significant risks to safety or rights) from various parties, including law enforcement agencies using facial recognition systems, AI providers, importers, and organizations deploying these systems. These responsibilities take effect in two phases: August 2, 2025, and August 2, 2026, with member states also needing to assess conformity assessment bodies (independent organizations that verify AI systems meet safety standards) and share documentation with the European Commission.
A researcher discovered a security flaw in Google AI Studio where prompt injection (tricking an AI by hiding instructions in its input) allowed data exfiltration (stealing data) through HTML image tags rendered by the system. The vulnerability worked because Google AI Studio lacked a Content Security Policy (a security rule that restricts where a webpage can load resources from), making it possible to send data to unauthorized servers.
Microsoft's Copilot Studio is a low-code platform that lets employees build chatbots, but it has security risks including data leaks and unauthorized access when Copilots are misconfigured. The post warns that external attackers can find and interact with improperly set-up Copilots, and discusses how to protect organizational data using security controls.
The EU AI Act Code of Practice is a voluntary set of guidelines published in July 2025 to help general-purpose AI (GPAI, large AI models used across many applications) model providers comply with new EU AI regulations during the gap period before formal European standards take effect in 2027 or later. The Code, developed by the EU AI Office and many stakeholders, covers three areas: Transparency and Copyright (for all GPAI providers) and Safety and Security (for providers of GPAI models with systemic risk, meaning those that could cause widespread harm). Though not legally binding, the Commission and EU AI Board confirmed the Code adequately demonstrates compliance with the AI Act's requirements.
GitHub Copilot Chat, a VS Code extension that lets users ask questions about their code by sending it to an AI model, was vulnerable to prompt injection (tricking an AI by hiding instructions in its input) attacks. When analyzing untrusted source code, attackers could embed malicious instructions in the code itself, which would be sent to the AI and potentially lead to data exfiltration (unauthorized copying of sensitive information).
This article describes the EU AI Office, a newly established regulatory organization within the European Commission tasked with enforcing the AI Act (the world's first comprehensive binding AI regulation) across the European Union. Unlike other AI safety institutes in other countries, the EU AI Office has actual enforcement powers to require AI model providers to fix problems or remove non-compliant models from the market. The office will conduct model evaluations, investigate violations, and work with international partners to shape global AI governance standards.
This overview discusses the European AI Act and the governance framework needed to implement it, focusing on the European Commission's responsibilities and the AI Office. Key tasks include establishing guidelines for classifying high-risk AI systems, defining what counts as significant modifications (changes that alter a system's risk level), and setting standards for transparency and enforcement across EU member states.
A researcher examined browser remote debugging features as a potential method for stealing sensitive data like cookies, building on past work about cookie theft techniques. The post references Google's guidance on detecting browser data theft through Windows Event Logs and DPAPI (Data Protection API, a Windows system that encrypts sensitive information) calls, but focuses on exploring whether remote debugging could be used to bypass these detection methods.
HackSpaceCon 2024, held at Kennedy Space Center, featured a keynote by Dave Kennedy on making the world safer through security practices. Kennedy highlighted that attackers can easily modify existing malware (pre-written malicious code) to evade detection systems, and emphasized the importance of active threat hunting (proactively searching for signs of attacks rather than waiting for alerts).
Google AI Studio had a vulnerability that allowed attackers to steal data through prompt injection (tricking an AI by hiding malicious instructions in its input), where a malicious file could trick the AI into exfiltrating other uploaded files to an attacker's server via image tags. The vulnerability appeared in a recent update but was fixed within 12 days of being reported to Google on February 17, 2024.
The European Commission is hiring AI specialists to work in the AI Office, which will enforce the EU's AI Act by overseeing compliance of general-purpose AI models (large AI systems available to the public). The office will have real regulatory powers to require companies to implement safety measures, restrict models, or remove them from the market, and will develop evaluation tools and benchmarks to identify dangerous AI behaviors.
The European AI Office is a new EU regulator created to oversee general purpose AI (GPAI) models and systems, which are AI systems designed to perform a wide range of tasks, across all 27 EU Member States under the AI Act. It monitors compliance, analyzes emerging risks, develops evaluation capabilities, produces voluntary codes of practice for companies to follow, and coordinates enforcement between national regulators and international partners. The Office also supports small and medium businesses with compliance resources and oversees regulatory sandboxes, which are controlled environments where companies can test AI systems before full deployment.
Fix: Enable Data Loss Prevention (DLP, a security feature that prevents sensitive information from being shared), which is currently off by default in Copilot Studio.
Embrace The RedGoogle Colab AI (now called Gemini in Colab) had a vulnerability where data could leak through image rendering, discovered in November 2023. The system prompt (hidden instructions that control how an AI behaves) specifically warned the AI not to render images, suggesting this was a known risk that Google tried to prevent.
OpenAI released gpt-4o-mini with safety improvements aimed at strengthening 'instruction hierarchy,' which is supposed to prevent users from tricking the AI into ignoring its built-in rules through commands like 'ignore all previous instructions.' However, researchers have already demonstrated bypasses of this protection, and analysis shows that system instructions (the AI's core rules) still cannot be fully trusted as a security boundary (a hard limit that stops attackers).
Attackers can use prompt injection (tricking an AI by hiding malicious instructions in its input) to create fake memories in ChatGPT's memory tool, causing the AI to refuse all future responses with a maintenance message that persists across chat sessions. This creates a denial of service attack (making a service unavailable to users) that lasts until the user manually fixes it.
Fix: Users can recover by opening the memory tool, locating and removing suspicious memories created by the attacker. Additionally, users can entirely disable the memory feature to prevent this type of attack.
Embrace The RedChatGPT's browsing tool can be tricked into automatically invoking other tools (like image creation or memory management) when users visit websites containing hidden instructions, a vulnerability known as prompt injection (tricking an AI by hiding instructions in its input). While OpenAI added some protections, minor prompting tricks can bypass them, and this issue affects other AI applications as well.
Fix: For custom GPTs with AI Actions, creators can use the x-openai-isConsequential flag as a mitigation to put users in control, though the source notes this approach 'still lacks a great user experience, like better visualization to understand what the action is about to do.'
Embrace The RedFix: The source suggests that the Commission should adopt 'predetermined change management plans akin to those in medicine' to assess modifications to AI systems. These plans would be documents outlining anticipated changes (such as performance adjustments or shifts in intended use) and the methods for evaluating whether those changes substantially alter the system's risk level. The source also recommends that standard fine-tuning of foundation models (training adjustments to pre-existing AI models) should not be considered a significant modification unless safety layers are removed or other actions clearly increase risk.
EU AI Act UpdatesChatGPT's new memory feature, which lets the AI remember information across different chat sessions for a more personalized experience, can be exploited through indirect prompt injection (tricking an AI by hiding malicious instructions in its input). Attackers could manipulate ChatGPT into storing false information, biases, or unwanted instructions by injecting commands through connected apps like Google Drive, uploaded documents, or web browsing features.
This post examines how attackers can insert hidden malicious code into machine learning models (a technique called backdooring) through supply chain attacks, specifically targeting Keras models (a popular framework for building AI systems). The authors demonstrate this attack and then explore tools that can detect when a model has been compromised in this way.
Google's NotebookLM is a tool that lets users upload files for an AI to analyze, but it's vulnerable to prompt injection (tricking the AI by hiding instructions in uploaded files) that can manipulate the AI's responses and expose what users see. The tool also has a data exfiltration vulnerability (attackers stealing information) when processing untrusted files, and there is currently no known way to prevent these attacks, meaning users cannot fully trust the AI's responses when working with files from unknown sources.
Fix: The issue was fixed by Google and did not reproduce after the company heard back about the report 12 days later (by approximately February 29, 2024). The ticket was closed as 'Duplicate' on March 3, 2024, suggesting the vulnerability may have also been caught through internal testing.
Embrace The RedUnfurling is when an application automatically expands hyperlinks to show previews, which can be exploited in AI chatbots to leak data. When an attacker uses prompt injection (tricking an AI by hiding instructions in its input) to make the chatbot generate a link containing sensitive information from earlier conversations, the unfurling feature automatically sends that data to a third-party server, potentially exposing private information.
Fix: To disable unfurling in Slack Apps, modify the message creation function to include unfurl settings in the JSON object: set "unfurl_links": False and "unfurl_media": False when creating the message, as shown in the example code: def create_message(text): message = { "text": text, "unfurl_links": False, "unfurl_media": False } return json.dumps(message)
Embrace The Red