AI Sec Watch

Citrini Research published a scenario describing how AI agents (autonomous AI systems that can make decisions and take actions independently) could trigger economic collapse by replacing white-collar workers with cheaper AI alternatives, creating a negative feedback loop where job losses reduce consumer spending, forcing companies to invest even more in AI to survive. The scenario imagines unemployment doubling and stock market value falling by a third within two years, though the researchers present it as a thought experiment rather than a prediction.

The U.S. Defense Secretary is meeting with Anthropic's CEO to pressure the company into allowing military use of Claude (Anthropic's AI system) for mass surveillance and autonomous weapons (weapons that can fire without human approval). Anthropic has refused these uses, and the Pentagon is threatening to label it a "supply chain risk" (a designation that would ban it from government contracts), which could void their $200 million military contract and force other Pentagon partners to stop using Claude.

OpenAI announced partnerships with four major consulting firms (Accenture, Boston Consulting Group, Capgemini, and McKinsey) to help deploy its enterprise AI platform called Frontier, which acts as an intelligence layer that connects different systems and data within organizations to help companies manage and build AI agents (tools that can independently complete tasks). These consulting partnerships aim to accelerate AI adoption for enterprise customers by combining OpenAI's technology with the consulting firms' existing relationships and deep knowledge of how businesses operate.

This newsletter covers multiple business and policy topics, including the Supreme Court striking down Trump's tariffs (duties, or taxes on imported goods) in a 6-3 decision, followed by Trump announcing a new 15% global tariff the next day. A major winter blizzard caused airlines to cancel 15% of U.S. flights on Monday, and Trump called on Netflix to fire board member Susan Rice.

This paper presents SIMix, a training framework for systems where multiple users learn AI models together over wireless networks while protecting their private data. The system uses Over-the-Air Mixup (OAM, a technique that combines data from multiple users through wireless transmission to hide sensitive information) and groups users strategically to reduce communication needs by up to 25% while defending against model inversion attacks (attempts to reconstruct private training data from a trained model) and label inference attacks (guessing what category a user's data belongs to).

Researchers developed PPOM-Attack, a method to fool face recognition (FR) systems by generating adversarial images (slightly altered photos that trick AI into misidentifying someone). Unlike earlier attacks that used substitute models (simpler AI systems trained to mimic the target system), PPOM-Attack directly queries the real face recognition system to learn how to create effective perturbations (tiny pixel changes), achieving 21.7% higher success rates while keeping the altered images looking natural.

Prompt injection attacks (tricking an AI by hiding malicious instructions in its input) pose a serious security risk to Large Language Models, as attackers can overwrite a model's original instructions to manipulate its responses. Researchers developed PromptFuzz, a testing framework that uses fuzzing techniques (automatically generating many variations of input data to find weaknesses) to systematically evaluate how well LLMs resist these attacks. Testing showed that PromptFuzz was highly effective at finding vulnerabilities, ranking in the top 0.14% of attackers in a real competition and successfully exploiting 92% of popular LLM-integrated applications tested.

Attackers are using autonomous AI agents (AI systems that can independently perform tasks without constant human direction) in supply chain attacks (compromises targeting the software or services that other programs depend on) to steal cryptocurrency from wallets. While this current campaign focuses on crypto theft, security researchers warn the technique could be adapted for much broader attacks.

As organizations deploy their own Large Language Models (LLMs), they are creating many internal services and APIs (application programming interfaces, which allow different software to communicate) to support them, but the real security risk comes from poorly secured infrastructure rather than the models themselves. Exposed endpoints (connection points where users, applications, or services communicate with an LLM) become attack vectors when they have excessive permissions and exposed long-lived credentials (authentication secrets that don't expire), allowing attackers far more access than intended. Endpoints typically become exposed gradually through small oversights during rapid deployment, such as APIs left publicly accessible without authentication, hardcoded tokens that are never rotated, or the false assumption that internal services are automatically safe.