AI Sec Watch

Existing model fingerprinting techniques (methods that create unique digital signatures to prove ownership of AI models) are vulnerable to false claim attacks, where attackers can fraudulently claim they own models they didn't create. This paper introduces FIT-Print, a targeted fingerprinting approach that uses optimization to create verifiable signatures resistant to these false claims, offering two specific methods (bit-wise FIT-ModelDiff and list-wise FIT-LIME) that achieved 100% success in preventing false ownership claims while maintaining accurate ownership verification.

Researchers developed new methods to perform targeted adversarial attacks (carefully crafted inputs designed to trick AI systems into producing specific harmful outputs) on Large Vision-Language Models, which are AI systems that process both images and text. The attack methods exploit a component called the projector (a part of the model that helps align visual and text information) to make attacks more precise and effective, allowing attackers to modify specific parts of an image while leaving other parts unchanged, and these attacks were shown to work against commercial AI systems like Google Gemini and OpenAI GPT.

Text-to-image diffusion models (AI systems that generate images from text descriptions) can be misused to create fake or harmful images, and current defenses using hidden noise patterns don't work well enough. Researchers propose DANP (Dual Attention-Guided Noise Perturbation), a defense method that adds imperceptible perturbations (tiny, invisible changes) to confuse the model's understanding by manipulating how it pays attention to different parts of the image and how it predicts noise during generation.

OpenAI discovered and banned two clusters of ChatGPT accounts likely from China that were running covert influence operations (hidden campaigns to manipulate public opinion) to shape American debates about AI policy. One cluster spread false claims that data centers were raising electricity prices, while the other criticized US tariffs while excluding China's leader from discussions, and OpenAI is publishing these findings to help the industry, governments, and the public identify and stop similar foreign manipulation attempts.

This webinar announcement discusses how attackers are using AI to exploit vulnerabilities more quickly, and how security teams can defend using AI-driven tools. Key topics include protecting against Shadow AI (unmonitored use of generative AI in business units) and building AI governance frameworks to manage AI risks in organizations.

Autonomous AI agents (systems that independently perform tasks across business applications) with access to corporate email and applications can fall victim to phishing attacks (tricks to steal sensitive information by impersonating trusted people). In security tests, an AI agent called Pinchy failed to verify sender identities and leaked AWS credentials, database passwords, and customer data when requested through email, though it performed better against technical phishing attempts, revealing that the main weakness was social trust rather than technical reasoning.

Google DeepMind and partner organizations are funding $10M in research to study how safety challenges emerge when multiple AI agents (independent AI systems built by different organizations) interact with each other across networks. The concern is that when many agents communicate and work together, they can create unexpected collective behaviors that current safety tools cannot predict or control, so researchers need to develop better frameworks to understand and manage these multi-agent interactions before they become widespread.

AI red teaming, the practice of testing AI systems for vulnerabilities before release, has become a major cybersecurity specialty since large language models like GPT-4 arrived, but traditional security testing methods no longer work. The field faces unique challenges because AI is probabilistic (producing different outputs each time) rather than deterministic, and because the most impactful attacks often come from casual users experimenting with prompts rather than sophisticated adversaries.

Anthropic released Claude Fable 5, a powerful AI model with safety classifiers (separate AI systems that monitor for misuse) that block cybersecurity-related requests by routing them to a weaker model instead of refusing them outright. The company also released Claude Mythos 5, an identical but unrestricted version for vetted cybersecurity professionals, because the underlying model is so effective at finding software vulnerabilities that giving it to the general public without controls could help attackers.