AI Sec Watch

Microsoft created a lightweight scanner that can detect backdoors (hidden malicious behaviors) in open-weight LLMs (large language models that have publicly available internal parameters) by identifying three distinctive signals: a specific attention pattern when trigger phrases are present, memorized poisoning data leakage, and activation by fuzzy triggers (partial variations of trigger phrases). The scanner works without needing to retrain the model or know the backdoor details in advance, though it only functions on open-weight models and works best on trigger-based backdoors.

Researchers have released new work on detecting backdoors (hidden malicious behaviors embedded in a model's weights during training) in open-weight language models to improve trust in AI systems. A backdoored model appears normal most of the time but changes behavior when triggered by a specific input, like a hidden phrase, making detection difficult. The research explores whether backdoored models show systematic differences from clean models and whether their trigger phrases can be reliably identified.

This research presents a semisupervised domain adaptation method (SDM), which helps AI classifiers work better when transferring knowledge from one data domain to another (like using a model trained on one type of data to work with a different type). The method addresses two main problems: limited labeled training data in the target domain and distribution divergence (differences in data patterns between source and target domains) by iteratively updating training data while balancing multiple objectives like structural risk and manifold consistency (geometric patterns in data).

This research presents an evidential deep neural network (EDNN), which is a machine learning model that combines evidence theory (a method for handling uncertainty) with convolutional neural networks (CNNs, algorithms that process images). Unlike traditional classifiers that assume all possible categories are known, the EDNN works under an open-world assumption (acknowledging that unknown categories may exist) and can classify items into single categories, multiple possible categories, or identify them as completely novel.

This article presents a method for teaching AI systems to control complex machinery safely and efficiently by combining optimal control (finding the best way to manage a system), safety constraints (hard limits that must never be broken), and a self-triggered mechanism (a system that only updates calculations when necessary). The approach uses a safety-filter structure to balance the competing goals of optimal performance and guaranteed safety, while reducing the computing power needed.

X's French offices were raided by Paris prosecutors investigating suspected illegal data extraction and possession of child sexual abuse material (CSAM, images depicting the sexual abuse of children), while the UK's Information Commissioner's Office launched a separate investigation into Grok (Elon Musk's AI chatbot) for its ability to create harmful sexualized images and videos without people's consent. The investigations were triggered by reports that Grok generated sexual deepfakes (fake sexual images created using real photos of women without permission) that were shared on X.

Claude Code is an agentic coding tool (software that can automatically write and execute code) that had a vulnerability in versions before 2.0.72 where attackers could bypass safety confirmation prompts and execute untrusted commands through the find command by injecting malicious content into the tool's context window (the input area where the AI reads information). The vulnerability has a CVSS score (a 0-10 severity rating) of 7.7, meaning it is considered high severity.

Claude Code, an agentic coding tool (AI software that writes and manages code), had a vulnerability in versions before 2.0.74 where a flaw in how it validated Bash commands (a Unix shell language) allowed attackers to bypass directory restrictions and write files outside the intended folder without permission from the user. The attack required the user to be running ZSH (a different Unix shell) and to allow untrusted content into Claude Code's input.

Claude Code, a tool that helps AI write and execute code automatically, had a security flaw before version 1.0.111 where it didn't properly check website addresses (URLs) before making requests to them. The app used a simple startsWith() check (looking only at the beginning of a domain name), which meant attackers could register fake domains like modelcontextprotocol.io.example.com that would be mistakenly trusted, allowing the tool to send data to attacker-controlled sites without the user knowing.