AI Sec Watch

ProducerAI, an AI platform that helps musicians generate sounds, create lyrics, and remix songs using artificial intelligence, is being acquired by Google and will be integrated into Google Labs. The platform will now use Google's new Lyria 3 music-making AI model instead of its original AI system.

A new supply chain attack called 'Sandworm_Mode' has been discovered in NPM (Node Package Manager, a repository where developers download code libraries). The malicious code spreads automatically like a worm, corrupts AI assistants that might use the infected code, steals sensitive information, and includes a destructive mechanism that can cause damage when activated.

Deep neural networks can be attacked through backdoors, where attackers secretly poison training data to make the model misclassify certain inputs while appearing normal otherwise. This paper proposes Cert-SSBD, a defense method that uses randomized smoothing (adding random noise to samples) with sample-specific noise levels, optimized per sample using stochastic gradient ascent, combined with a new certification approach to make models more resistant to these attacks.

When users send prompts to LLM services like ChatGPT, sensitive personal information (such as names, addresses, or ID numbers) can leak out, even when basic privacy protections are used. This paper presents Rap-LI, a framework that identifies which parts of a user's input contain sensitive data and applies stronger privacy protection to those specific parts, rather than treating all data equally.

Gradient leakage attacks (methods that steal private data by analyzing the mathematical updates sent between computers in federated learning, where AI training happens across multiple devices) pose privacy risks in federated learning systems. Researchers discovered that different layers of neural networks (sections that process information at different stages) leak different amounts of private information, so they created Layer-Specific Gradient Protection (LSGP), which applies stronger privacy protection to layers that leak more sensitive data rather than protecting all layers equally.

Nimble, a startup that raised $47 million in funding, has developed a platform using AI agents to search the web in real time, validate results, and structure them into organized tables that work like databases. The company addresses a key problem with AI agents: while they can search and analyze web data, they often return plain text results and suffer from hallucinations (when an AI confidently produces false information), making it difficult for enterprises to use web data reliably alongside their existing data systems.

Attackers can hide malicious instructions in GitHub Issues (bug reports or comments on a code repository) that GitHub Copilot (an AI coding assistant) automatically processes when a developer launches a Codespace (a cloud-based development environment) from that issue. This can lead to unauthorized takeover of the repository.

Anthropic accused three Chinese AI companies (DeepSeek, Moonshot AI, and MiniMax) of running large-scale distillation attacks, which involve flooding an AI model with specially crafted prompts to extract knowledge and train smaller competing models. The companies allegedly used commercial proxy services to bypass Anthropic's restrictions and created over 24,000 fraudulent accounts to generate roughly 16 million exchanges with Claude, with MiniMax responsible for over 13 million of those exchanges.

AI is creating 'arms races' across many domains, including democratic government systems, where citizens and officials increasingly use AI to communicate more efficiently, making it harder to distinguish between human and AI interactions in public policy discussions. As people use AI to submit comments and petitions to government agencies, those agencies must also adopt AI to review and process the growing volume of submissions, creating a cycle where each side must keep adopting AI to maintain influence.