AI Sec Watch

AI agents are now being deployed in companies to automate business workflows, such as managing customer renewal requests by reading emails, accessing CRM (customer relationship management, a database of customer information) data, and taking actions like drafting responses and updating records. Unlike simple text generators, these agents actively read sensitive business data, use system credentials (login information that grants access), and call external tools, which creates new security challenges that organizations need to address.

This research proposes TAPGuard, a framework for detecting cascading threats in Trigger-Action Programming (TAP, a system where one event automatically triggers another action, commonly used in smart home devices). The framework uses large language models (AI systems trained on text) to understand the semantic meaning (the actual intent and meaning, not just the structure) of automation rules and identifies two types of threats: explicit ones from direct device interactions and implicit ones from rules sharing environmental variables that shouldn't interact. TAPGuard performs better than existing methods at catching these dangerous rule combinations.

Researchers created SOOM, a defense method that obfuscates (hides or disguises) deep learning operators to protect against model extraction attacks, where attackers reverse-engineer compiled neural network code to recreate trainable models. Built on TVM (a deep learning compiler), SOOM uses a machine learning cost model to scramble how operators work while keeping inference fast, achieving a 89% failure rate against extraction attacks with minimal performance slowdown.

Existing model fingerprinting techniques (methods that create unique digital signatures to prove ownership of AI models) are vulnerable to false claim attacks, where attackers can fraudulently claim they own models they didn't create. This paper introduces FIT-Print, a targeted fingerprinting approach that uses optimization to create verifiable signatures resistant to these false claims, offering two specific methods (bit-wise FIT-ModelDiff and list-wise FIT-LIME) that achieved 100% success in preventing false ownership claims while maintaining accurate ownership verification.

Researchers developed new methods to perform targeted adversarial attacks (carefully crafted inputs designed to trick AI systems into producing specific harmful outputs) on Large Vision-Language Models, which are AI systems that process both images and text. The attack methods exploit a component called the projector (a part of the model that helps align visual and text information) to make attacks more precise and effective, allowing attackers to modify specific parts of an image while leaving other parts unchanged, and these attacks were shown to work against commercial AI systems like Google Gemini and OpenAI GPT.

Text-to-image diffusion models (AI systems that generate images from text descriptions) can be misused to create fake or harmful images, and current defenses using hidden noise patterns don't work well enough. Researchers propose DANP (Dual Attention-Guided Noise Perturbation), a defense method that adds imperceptible perturbations (tiny, invisible changes) to confuse the model's understanding by manipulating how it pays attention to different parts of the image and how it predicts noise during generation.

OpenAI discovered and banned two clusters of ChatGPT accounts likely from China that were running covert influence operations (hidden campaigns to manipulate public opinion) to shape American debates about AI policy. One cluster spread false claims that data centers were raising electricity prices, while the other criticized US tariffs while excluding China's leader from discussions, and OpenAI is publishing these findings to help the industry, governments, and the public identify and stop similar foreign manipulation attempts.

This webinar announcement discusses how attackers are using AI to exploit vulnerabilities more quickly, and how security teams can defend using AI-driven tools. Key topics include protecting against Shadow AI (unmonitored use of generative AI in business units) and building AI governance frameworks to manage AI risks in organizations.

Autonomous AI agents (systems that independently perform tasks across business applications) with access to corporate email and applications can fall victim to phishing attacks (tricks to steal sensitive information by impersonating trusted people). In security tests, an AI agent called Pinchy failed to verify sender identities and leaked AWS credentials, database passwords, and customer data when requested through email, though it performed better against technical phishing attempts, revealing that the main weakness was social trust rather than technical reasoning.