AI Sec Watch

This research addresses the risk that AI models trained on private data could leak sensitive information if attackers extract data from them. The authors propose a method called differentially private synthetic distillation, which converts a trained model into a privacy-protected version without needing access to the original private data, using a generator to create synthetic data and noise to obscure sensitive patterns.

This research introduces DeSA, a protocol for secure aggregation (a privacy technique that protects individual data while combining results) in federated learning (a machine learning approach where multiple devices train a shared model without sending raw data to a central server) across decentralized device-to-device networks. The protocol addresses challenges in zero-trust networks (environments where no participant is automatically trusted) by using zero-knowledge proofs (cryptographic methods that verify information is correct without revealing the information itself) to verify model training, protecting against Byzantine attacks (attacks where malicious nodes send false information to disrupt the system), and employing a one-time masking method to maintain privacy while allowing model aggregation.

Subgraph Federated Learning (FL, a system where pieces of a graph are distributed across multiple devices to protect data privacy) is vulnerable to backdoor attacks (hidden malicious functions that cause a model to behave incorrectly when triggered). Researchers developed BEEF, an attack method that uses adversarial perturbations (carefully crafted small changes to input data that fool the model) as hidden triggers while keeping the model's internal parameters unchanged, making the attack harder to detect than existing methods.

Researchers discovered that speech translation (ST) systems, which convert spoken words from one language to another, can be tricked by specially crafted audio manipulations that are imperceptible to human ears. They demonstrated two attack methods: adapting techniques from ASR (automatic speech recognition) attacks and using music-based perturbations to guide the system toward producing harmful outputs. These attacks worked across multiple languages and models, revealing a fundamental weakness in how current speech translation systems process and understand audio.

vLLM, a system for running and serving large language models, has a Server-Side Request Forgery vulnerability (SSRF, where an attacker tricks a server into making requests to unintended targets) in its multimodal feature before version 0.14.1. The bug exists because two different Python libraries interpret backslashes differently, allowing attackers to bypass security checks and force the vLLM server to send requests to internal network systems, potentially stealing data or causing failures.

PyTorch (a Python package for tensor computation) versions before 2.10.0 have a vulnerability in the `weights_only` unpickler that allows attackers to create malicious checkpoint files (.pth files, which store model data) triggering memory corruption and potentially arbitrary code execution (running attacker-chosen commands) when loaded with `torch.load(..., weights_only=True)`. This is a deserialization vulnerability (a weakness where loading untrusted data can be exploited).

China's DeepSeek AI tool, which caused significant market disruption when it launched a year ago, is now being adopted by an increasing number of US companies. The episode discusses this growing trend of Chinese AI technology being integrated into American business operations.

The White House digitally altered a photograph of an activist's arrest by darkening her skin and distorting her facial features to make her appear more distraught than in the original image posted by the Department of Homeland Security. AI detection tools confirmed the manipulation, raising concerns about how generative AI (systems that create images from text descriptions) and image editing technology can be misused by government to spread false information and reinforce racial stereotypes. The incident highlights the danger of deepfakes (realistic-looking fake media created with AI) and the importance of protecting citizens' right to independently document government actions.

AnythingLLM is an application that lets users feed documents into an LLM so it can reference them during conversations. Versions before 1.10.0 had a security flaw where an API key (QdrantApiKey) for Qdrant, the database that stores document information, could be exposed to anyone without authentication (credentials). If exposed, attackers could read or modify all the documents and knowledge stored in the database, breaking the system's ability to search and retrieve information correctly.