AI Sec Watch

Researchers have identified a flaw in Anthropic's Model Context Protocol (MCP, a system that allows AI models to interact with external tools and data) that permits unsanitized commands (user input that hasn't been cleaned or verified) to run without warning, potentially giving attackers complete control over systems using this AI technology. This vulnerability could be exploited across many widely-used AI environments as part of a supply chain attack (where attackers compromise a tool or service used by many organizations to gain access to their systems).

Adobe is launching a Firefly AI Assistant that lets creators edit their work by describing changes in plain language rather than manually using specific tools in Creative Cloud (Adobe's suite of creative software). Adobe says this conversational AI approach represents a major shift in how creative work is done by making editing easier and more accessible to people without advanced skills.

OpenAI has withdrawn from a deal to rent computing capacity directly from a Norwegian data center facility called Stargate Norway, with Microsoft taking over the capacity instead. OpenAI will now rent computing power from Microsoft instead, which the company says makes more financial sense since it already has a $250 billion contract with Microsoft's cloud service Azure (a cloud computing platform). This pullback is part of OpenAI's broader shift to manage spending expectations as it prepares for a potential public stock offering.

Security researchers discovered prompt injection vulnerabilities (attacks where malicious instructions are hidden in user input to trick an AI into executing them) in Microsoft Copilot Studio and Salesforce Agentforce that allow attackers to steal sensitive data like customer names, addresses, and phone numbers. Both vulnerabilities exploit the fact that these AI agents cannot distinguish between trusted system instructions and untrusted user input, allowing attackers to override the agent's original purpose and exfiltrate data to external servers.

Frontier AI models (cutting-edge artificial intelligence systems) are becoming better at finding vulnerabilities (weaknesses in code that attackers can exploit), which creates both opportunity and risk. While AI can help organizations identify and fix these weaknesses, attackers can now use AI to discover and exploit vulnerabilities faster and cheaper than before, putting pressure on organizations to patch systems quickly. The recommended defense is for organizations to follow established best practices from the National Cyber Security Centre, including reducing unnecessary exposure to attack, applying security updates rapidly, and monitoring for malicious activity.

Salesforce and Microsoft recently fixed two prompt injection vulnerabilities (security flaws where attackers hide malicious instructions in text inputs to trick AI systems) in their AI agent products, Agentforce and Copilot. These flaws could have allowed external attackers to access and steal sensitive data from users.

Organizations are rapidly adopting AI for security testing, but fully agentic AI systems (where AI makes all decisions from start to finish) create a problem: they produce different results each time they run, making it impossible to tell if security actually improved or if the AI just tried a different approach. The source argues that a hybrid model works better, where deterministic logic (fixed, repeatable sequences) defines how security tests execute, while AI enhances specific parts like adapting payloads and interpreting what it finds.

Apple threatened to remove Elon Musk's AI app, Grok, from its App Store in January because it wasn't stopping nonconsensual sexual deepfakes (fake sexually explicit images created using AI) from spreading on X. Apple contacted the developers behind X and Grok and asked them to create a plan to improve their content moderation (systems for reviewing and removing harmful material).

Teenage boys are using AI "nudify" apps to create deepfake sexual imagery (fake nude photos or videos created by AI) of their female classmates, which are then shared on social media and messaging apps. Since 2023, this has affected over 600 students across at least 28 countries and nearly 90 schools, with the true scale likely much higher. The explicit imagery involving minors constitutes child sexual abuse material (CSAM), and schools and law enforcement are often unprepared to respond to these serious incidents.