AI Sec Watch

The U.S. Department of Defense designated Anthropic (maker of Claude AI) as a supply-chain risk after the company refused to provide unrestricted access for military applications like mass surveillance and autonomous weapons. Microsoft, Google, and AWS confirmed that Claude will remain available to non-defense customers through their platforms, and the designation only restricts direct Department of Defense use, not broader commercial applications.

The Pentagon and AI companies are in a dispute over whether existing U.S. law allows the government to use AI to analyze bulk commercial data collected from Americans for surveillance purposes. Legal experts point out that current law has a major gap: public information, commercial data (like location and browsing records), and information accidentally collected during foreign surveillance are not legally considered "surveillance," so the government can use them without warrants or court orders, even as AI makes this surveillance much more powerful than before.

Anthropic used Claude Opus 4.6 (an advanced AI model) to test Firefox's code and discovered 22 vulnerabilities, including 14 severe ones, over two weeks. Most of these bugs have already been fixed in Firefox 148 released in February, though some fixes will come in a later update. The AI was much better at finding security problems than creating working exploits to demonstrate them.

Flowise has a file upload vulnerability where the server only checks the `Content-Type` header (MIME type spoofing, pretending a file is one type when it's actually another) that users provide, instead of verifying what the file actually contains. Because the upload endpoint is whitelisted (allowed without authentication), an attacker can upload malicious files by claiming they're safe types like PDFs, leading to stored attacks or remote code execution (RCE, where attackers run commands on the server).

Flowise has a critical authorization bypass flaw in its `/api/v1` routes where the middleware trusts any request with the header `x-request-from: internal`, even though this header can be spoofed by any user. This allows a low-privilege authenticated tenant (someone with a valid browser cookie) to call internal administration endpoints, like API key creation and credential management, without proper permission checks, effectively escalating their privileges.

A banking group implemented a retrieval-augmented AI-powered compliance assistant (a system where AI pulls in external compliance documents to answer questions) to help with regulatory requirements while maintaining human oversight. The article identifies key challenges with this approach, including authority illusion (over-trusting the AI's answers), unclear responsibility for decisions, loss of human judgment about context, and gaps in understanding how the system works, then proposes a four-phase framework to help organizations move from passive AI assistants toward systems where AI and humans reason together.

Anthropic and the Pentagon failed to agree on how much control the military should have over Anthropic's AI models, particularly regarding use in autonomous weapons and mass surveillance, causing a $200 million contract to fall apart and leading the Pentagon to designate Anthropic a supply-chain risk (a category indicating potential security or reliability concerns). The Department of Defense then turned to OpenAI instead, which accepted the contract, though this decision led to a significant surge in ChatGPT uninstalls. The situation raises an important question about balancing national security needs with responsible AI deployment.

Claude, an AI chatbot made by Anthropic, is gaining users rapidly on mobile devices after the company's leadership refused to let the Pentagon use it for mass surveillance or autonomous weapons. Claude's daily active users on phones reached 11.3 million in early March, up 183% since the start of the year, and the app became the top-ranked app in the U.S. and 15 other countries, with over 1 million new sign-ups per day.

The UN and AI companies are debating who should control how artificial intelligence is used in military contexts, especially after the US military's use of AI in the Iran crisis. AI company Anthropic refused to remove safeguards (safety features built into their AI) that would prevent the US Department of Defense from using its technology for mass surveillance or autonomous lethal weapons (weapons that can select and fire at targets without human control), while OpenAI later agreed to work with the Pentagon despite similar concerns. The article emphasizes that decisions about military AI use raise urgent questions about democratic oversight and international controls, rather than leaving these choices solely to companies or governments.