AI Sec Watch

Anthropic is giving the European Union access to Mythos, its most advanced AI model, after months of requests due to cybersecurity concerns. Mythos excels at finding security flaws in software (vulnerabilities, or weaknesses in code), but officials worry bad actors could misuse it to accelerate cybercrimes by exploiting thousands of previously unknown weaknesses. The EU is still working out the exact terms of the deal and discussing AI risks with partner countries.

Microsoft is holding its Build developer conference to showcase new AI capabilities and rebuild trust with developers, who have lost confidence in Windows and GitHub. The company plans to announce new AI models integrated into Windows, a new reasoning model (an AI system designed to work through complex problems step-by-step), and a Copilot super app (a unified interface for multiple AI assistant features).

AI-generated music is becoming widespread in the music industry, with over 50,000 AI-generated songs uploaded daily to streaming platforms, making it harder to identify and filter out. The Recording Academy, which runs the Grammy Awards, currently has rules that exclude AI music from eligibility, but the CEO acknowledges that AI tools like Suno are now omnipresent in music production sessions.

The `PATCH /workspaces/{id}` endpoint in praisonai-platform allows any workspace member to change the workspace's name, description, and settings (a free-form JSON configuration object) because it only checks that the user is a member, not that they have owner-level permissions. This is dangerous because attackers could inject malicious settings that could redirect API calls to attacker-controlled servers, disable logging, or change other critical configurations depending on what the platform reads from the settings field.

This paper presents a mathematical framework for making cybersecurity decisions when facing uncertain threats in turn-based security games (scenarios where defenders and attackers take turns making moves). The framework handles uncertainty about both what the attacker might do and how well defensive controls will work, using game theory (the study of strategic decision-making between competing parties) and robust optimization (techniques for finding solutions that work well even when future conditions are unknown). The researchers show their approach outperforms traditional defensive strategies and demonstrate its usefulness through a network attack example.

Large language models raise privacy concerns because the knowledge they learn becomes deeply entangled in their structure, making it hard to make them "forget" specific information. Researchers developed a privacy-preserving machine unlearning method (a technique to remove learned data from AI models) that eliminates high-risk information from model outputs and uses differentially-private randomization (adding statistical noise to hide sensitive data) to ensure unlearned information cannot be identified, without requiring model parameter adjustments.

This research introduces QAFD (Quality-Assisted Forgery Detection), a new system for detecting AI-generated images by analyzing both visual features and quality-related artifacts that different generative models produce. The system uses a quality-guided approach to help AI models better understand degradation patterns in fake images, allowing it to detect AI-generated content more reliably even when tested on unseen generative models and images that have been edited after creation.

TrustSearch is a system that performs reverse image search (finding similar images in a database) on encrypted data stored in the cloud while protecting user privacy. It uses Intel SGX (a trusted execution environment, which is a secure area on a processor where sensitive operations can run protected from outside access) to search images without decrypting them, but had to optimize its design because SGX has limited memory and processing resources.

BadBone is a backdoor attack (a type of hidden vulnerability where an attacker secretly compromises a model to make it misbehave on specific tasks) that targets backbone models (large pre-trained neural networks that serve as the foundation for smaller AI systems) used in prompt learning (a technique where users guide AI behavior by providing example inputs called prompts). The attack is stealthy because it hides the backdoor in the backbone model rather than in the prompt learning process itself, so downstream tasks using prompt learning inherit the vulnerability while the model appears to work normally. Testing shows that current security defenses against backdoors are largely ineffective against BadBone, indicating the need for stronger protections.