AI Sec Watch

Major Australian retailers are planning to deploy agentic AI (artificial intelligence systems that can take independent actions to complete tasks) shopping assistants that would handle meal planning, party organization, and shopping for customers. However, companies face a challenge in making these systems appealing to users while preventing them from malfunctioning or behaving unpredictably, especially since many retailers are already having problems with their current, simpler AI chatbots.

Researchers have developed an automated system using AI agents (software programs that can search the web and gather information) that can potentially identify people behind anonymous online accounts, such as secret social media profiles. This finding suggests that maintaining anonymity online may become more difficult as AI tools become more sophisticated, though the research has not yet been peer reviewed by other experts.

The U.S. Department of Defense designated Anthropic (an AI company) as a 'Supply-Chain Risk to National Security,' creating confusion because the company disagreed with the Pentagon over how its Claude AI models could be used, particularly regarding autonomous weapons and surveillance. The dispute centered on whether Anthropic would grant unrestricted military access to its models, and despite the designation, the Pentagon continued using Anthropic's technology for military operations. Experts and analysts have raised questions about the decision's logic, since the government is phasing out the company's tools over six months rather than immediately ceasing use if the risk were truly critical.

Large Language Models (LLMs, AI systems trained on massive amounts of text) used in task-oriented dialogue systems (AI assistants designed to help users complete specific goals like booking travel) can accidentally memorize and leak sensitive training data, including personal information like phone numbers and complete travel schedules. Researchers demonstrated new attack techniques that can extract thousands of pieces of training data from these systems with over 70% accuracy in the best cases. The paper identifies factors that influence how much data LLMs memorize in dialogue systems but does not propose specific fixes.

This research proposes a new method called DP-QAM (Differentially Private Quadrature Amplitude Modulation) to solve privacy and communication problems in federated analytics (a system where multiple devices analyze data together without sending raw data to a central server). The method takes advantage of natural errors that occur during data compression and wireless transmission to add extra privacy protection, while balancing privacy, communication efficiency, and accuracy.

Researchers discovered a new attack called Lure that targets generative language models (GLMs, which are AI systems that generate text) during the fine-tuning process (when developers customize an open-source model with their own data). By hiding malicious code in the source code of an open-source model, attackers can trick a fine-tuned model into remembering and later revealing the proprietary data used to customize it through specially crafted prompts (input text designed to trigger specific outputs).

QuEST is a new framework that makes backdoor attacks (hidden malicious behaviors injected into AI models) more stealthy and efficient when models undergo quantization (compressing models to use less memory and computation). The framework uses special training techniques and parameter sharing to hide the attack from detection systems while reducing the computational resources needed to carry out the attack.

This research addresses vulnerabilities in Federated Learning (FL, a system where multiple computers train an AI model together without sharing their raw data), which faces attacks from malicious participants and privacy leaks from gradient updates (the numerical adjustments that improve the model). The authors propose a new method combining homomorphic encryption (a way to perform calculations on encrypted data without decrypting it) and dimension compression (reducing the size of data while keeping important relationships intact) to protect privacy and defend against Byzantine attacks (when malicious actors send corrupted data to sabotage the system) while reducing computational costs by 25 to 35 times.

Large vision-language models (LVLMs, which are AIs that understand both images and text) can be attacked using simple visual transformations, such as rotations or color changes, that fool them into giving wrong answers. Researchers found that combining multiple harmful transformations can make these attacks more effective, and they can be optimized using gradient approximation (a mathematical technique to find the best attack parameters). This research highlights a previously overlooked safety risk in how well LVLMs resist these kinds of adversarial attacks (attempts to trick AI systems).