AI Sec Watch

The `download_media` and `auth_fetch` tools in auth-fetch-mcp accept any URL without validation, allowing an attacker (via prompt injection or a malicious MCP client) to make the server fetch from private or internal services like cloud metadata endpoints or localhost, and then exfiltrate the response data. The `download_media` tool makes this worse by saving fetched content to disk where it can be read and stolen.

An attacker published malicious code in guardrails-ai version 0.10.1 on PyPI (a package repository where developers download Python libraries), but PyPI removed it within 2 hours and found no evidence that user data was stolen through this compromise. This is an example of a supply chain attack, where someone tries to harm users by corrupting a widely-used software package.

GitHub is replacing cash bounties with swag rewards for low-impact bug reports and asking researchers to stop submitting low-quality reports, because AI tools have flooded the platform with submissions that don't represent real security risks. The company clarified that many rejected reports describe scenarios where users must actively engage with malicious content (like cloning a malicious repository), which means the security boundary lies with the user's decision to trust that content rather than with GitHub's security controls.

Palladium is a system that protects private training data and model parameters when training deep neural networks (DNNs, AI systems with many layers that learn patterns from data) on remote cloud servers with GPUs. The system uses TEEs (trusted execution environments, secure areas of a processor that are isolated from the rest of the system) combined with a "Cloak" strategy to hide sensitive information while still allowing most computations to run on untrusted accelerators, achieving both privacy protection and reasonable performance.

Diffusion models (AI systems that generate images from text descriptions) can be misused to create unauthorized portraits or copies of artistic styles through personalization, which threatens privacy and copyright. PersGuard is a new defense framework that embeds protective backdoors (hidden mechanisms) into these models before release, so that if someone tries to personalize the model with protected images, it generates predetermined protective outputs instead, while still working normally for unprotected images.

Researchers developed a new method to attack 3D point cloud models (AI systems that process 3D sensor data) using only the final prediction label, without needing access to the model's internal details or calculations. Their technique uses a spectrum-aware decision boundary algorithm to create adversarial examples (slightly modified inputs that fool the AI) that are harder to detect and more practical to deploy in real-world scenarios than existing attack methods.

SilentNoise addresses a problem in differential privacy (DP, a method for analyzing data while protecting individual privacy), which traditionally relies on one trusted party holding all sensitive data, creating a security risk. The researchers propose a decentralized system using secure multiparty computation (MPC, where multiple parties jointly compute results without fully revealing their individual data) that allows noise (random data added for privacy) to be generated securely even when some parties act maliciously, improving both efficiency and accuracy compared to previous approaches.

Garland is a system for recommendation engines that use graph neural networks (GNNs, which are AI models that learn patterns from interconnected user-item relationships) in federated settings, where data stays on users' devices instead of being sent to one central server. The system addresses a key problem: untrusted servers that help expand users' local data can spy on both item information and user relationships, so Garland uses secret-shared shuffle (a cryptographic technique that mixes data while keeping it encrypted) to protect privacy while still catching if a malicious server tries to cheat.

This newsletter covers several AI industry developments, including Elon Musk losing his lawsuit against OpenAI (a company creating large language models, which are AI systems trained on large amounts of text data) because he sued too late under statutes of limitations rather than on the merits of whether OpenAI violated its nonprofit mission. Other stories include Anduril and Meta developing augmented-reality smart glasses (wearable devices that overlay digital information on the physical world) for military use with eye-tracking controls, and Google preparing to showcase its AI capabilities at its I/O developer conference while facing competition from other AI companies.