AI Sec Watch

Enterprise AI systems deployed for security work are heavily restricted by safety guardrails (automated filters designed to prevent harmful outputs), while attackers freely use jailbroken models (AI systems with safety measures bypassed), open-source alternatives, and purpose-built malicious tools. This creates an asymmetry where defenders face routine refusals when requesting legitimate defensive content like phishing simulations or proof-of-concept code, while attackers can easily circumvent safety measures through prompt injection (tricking AI by hiding instructions in its input) and other well-documented techniques, giving them a significant operational advantage.

Overseas 'content farms' based in Vietnam are using AI to create fake videos and images of UK politicians, spreading them on Facebook to go viral and potentially earn money through the platform's monetization program. The fake content, called deepfakes (digitally altered videos, pictures, or audio made to look real), depicts politicians in false situations like hospital stays or compromising scenarios, and Meta has removed some pages after investigation, though new ones continue appearing daily.

As generative AI (systems that create new content based on patterns in training data) becomes widespread across industries, organizations need specialized security tools to protect their AI infrastructure and data from cyber threats. AI Security Posture Management (AI-SPM) is a new category of security software designed to monitor, assess, and secure AI systems, complementing existing tools like CSPM (Cloud Security Posture Management, which protects cloud environments) and DSPM (Data Security Posture Management, which prevents data breaches).

More than 30 employees from OpenAI and Google DeepMind filed a court statement supporting Anthropic in a lawsuit against the U.S. Defense Department, which labeled the AI company a supply-chain risk after Anthropic refused to let the Pentagon use its technology for mass surveillance or autonomous weapons. The employees argue that the Pentagon could have simply canceled its contract with Anthropic and purchased from another AI company instead of designating it as a supply-chain risk, a label typically reserved for foreign adversaries. They contend that if the government is allowed to punish Anthropic this way, it will harm U.S. competitiveness in AI and discourage open discussion about the risks of AI systems.

AI chip technology is advancing faster than data centers can be built, creating a financial risk for companies like Oracle that are investing heavily in infrastructure. OpenAI has decided not to expand its partnership with Oracle's Texas data center because it wants access to newer Nvidia chips rather than the older generation (Blackwell processors) that will be ready in a year, highlighting how quickly AI hardware becomes outdated. This mismatch is particularly risky for Oracle, which is funding its $100 billion expansion primarily through debt rather than using cash from existing profitable businesses like its competitors do.

Anthropic, an AI company, filed a lawsuit against the Department of Defense after being labeled a supply chain risk (a government designation suggesting a company could threaten critical systems). Nearly 40 employees from competing AI companies OpenAI and Google, including prominent figures, filed a legal support document expressing concerns about this decision and its implications for AI technology.

Attackers are running a campaign called 'InstallFix' that uses malvertising (ads serving malware) combined with ClickFix tactics (fake warning popups that trick users into taking action) to direct people to fake websites pretending to be Claude, an AI coding assistant. The attack exploits how developers use AI tools and command-line interfaces (text-based programs that run on computers) to execute code.

The U.S. Defense Department banned Anthropic's AI models after a review by Pentagon technology leadership, designating the company a supply chain risk (a classification historically reserved for foreign adversaries) and requiring defense contractors to certify they don't use its technology. The decision surprised many officials who considered Anthropic's models superior and had deployed them in classified military networks, and defense experts worry it sets a troubling precedent while removing a trusted AI vendor that military personnel relied on.

vLLM has a bypass in its SSRF (server-side request forgery, where an attacker tricks a server into making requests to unintended targets) protection because the validation layer and the HTTP client parse URLs differently. The validation uses urllib3, which treats backslashes as literal characters, but the actual requests use aiohttp with yarl, which interprets backslashes as part of the userinfo section. An attacker can craft a URL like `https://httpbin.org\@evil.com/` that passes validation for httpbin.org but actually connects to evil.com.