AI Sec Watch

Anthropic lost a US Department of Defense contract after refusing to let the Pentagon use its AI models for mass surveillance or fully autonomous weapons (systems that make kill decisions without human input), while OpenAI secured the contract by agreeing to provide classified government systems with AI. The article argues this outcome may benefit Anthropic by reinforcing its brand as a trustworthy, ethical AI provider in a competitive market where different AI models perform similarly.

Threat actors are using AI and language models as operational tools to speed up cyberattacks across all stages, from creating phishing emails to generating malware code, while human attackers maintain control over targeting and deployment decisions. Emerging experiments with agentic AI (where models make iterative decisions with minimal human input) suggest attackers may develop more adaptive and harder-to-detect tactics in the future. Microsoft reports disrupting thousands of fraudulent accounts and partnering with industry to counter AI-enabled threats through technical protections and responsible AI practices.

GitHub Copilot CLI had a vulnerability where attackers could execute arbitrary code by hiding dangerous commands inside bash parameter expansion patterns (special syntax for manipulating variables). The safety system that checks whether commands are safe would incorrectly classify these hidden commands as harmless, allowing them to run without user approval.

OpenAI signed a deal with the U.S. Department of Defense to provide AI tools after rival Anthropic refused, sparking criticism and a 300% spike in ChatGPT uninstalls. The company added contract language stating the AI won't be used for domestic surveillance of U.S. citizens, but critics argue the agreement contains vague 'weasel words' (deliberately ambiguous phrases that allow one side to avoid accountability) like 'intentionally,' 'deliberately,' and 'unconstrained' that the government can interpret loosely to justify mass surveillance anyway.

Attackers are using InstallFix, a social engineering technique, to distribute the Amatera Stealer malware through fake installation pages for Claude Code that closely mimic the legitimate site. These cloned pages contain malicious install commands designed to trick users into running code that downloads the malware, and are promoted via malvertising (fake ads in search results) on Google Ads.

Cyberattackers used popular AI chatbots, specifically Anthropic's Claude and OpenAI's ChatGPT, along with a detailed instruction set (called a prompt), to break into Mexican government agencies and steal citizens' personal data. This incident demonstrates how AI tools can be misused by attackers to carry out coordinated cybercrimes against government systems.

Online ads are becoming a major way to spread malware (malicious software) into organizations, with malvertising (malware delivered through ads) now surpassing email and direct hacking as the top delivery method. AI is making this worse by enabling attackers to create adaptive malware that changes its behavior based on a user's location, browser, or device, allowing millions of infected ads to spread across websites in seconds.

Vertical split learning (VSL, a privacy method that divides an AI model between multiple clients and a server) has been found vulnerable to a new stealthy attack called TPA-VSL, where attackers manipulate the embedding model (the part that converts data into numerical vectors) to misclassify targeted samples without leaving obvious signs of poisoning. The attack uses diffusion models (AI systems that generate data by reversing a noise process) and special encoders to trick the system into mapping target data to wrong classes, achieving a 30% higher success rate than existing attacks.

Federated learning (a system where multiple parties train AI models together while keeping their data private) faces two main problems: model updates can leak sensitive information, and it's hard to detect poisoning attacks (when malicious participants deliberately corrupt the training process). ClusterGuard is a new secure aggregation protocol (a method for safely combining model updates from many participants) that uses clustering, masking techniques, and filtering mechanisms to protect privacy while detecting and resisting poisoning attacks, even when up to 20% of participants are malicious.