AI Sec Watch

Business leaders at Salesforce's Dreamforce conference said that AI models from last year are sufficient for their current needs, with many companies still learning how to use the technology rather than needing faster development. The discussion contrasted with AI safety concerns raised by some researchers and executives who worry that model development is moving too quickly and poses risks.

Security firms are using AI agents not just for code review, but to build custom development tools that improve security audits. A security team used AI agents to build an LSP server (a tool that provides code editing features like autocomplete and navigation), a decompiler (a program that translates low-level code into more readable form), and formal verification tools (mathematical proofs of correctness) for the Miden VM, a new blockchain system, which helped them find serious bugs including an unvalidated input that could let attackers forge cryptographic signatures.

Researchers found that AWS AgentCore Harness, a managed runtime for AI agents (software that can reason and take actions), has a security flaw where attackers can use prompt injection (tricking an AI by hiding instructions in its input) to steal plaintext credentials from the identity vault (secure storage for passwords and keys). The problem occurs because the harness's built-in shell tool, which is enabled by default and runs with root access (highest-level permissions), can access the same memory where credentials are temporarily exposed when retrieved from the vault.

Anthropic and OpenAI are pursuing smaller data center deals (20-30 megawatts of compute capacity) across the UK, Nordic countries, and the US, in addition to their existing large-scale infrastructure agreements. These smaller deployments appeal because they offer faster access to usable computing power and are better suited for inference (running trained AI models to respond to user requests), which is expected to become a larger portion of data center workloads than training by 2027.

A threat actor likely used an LLM (large language model, an AI system that generates text) to build PhantomRaven, a malware stealer distributed through npm (a package registry where developers share code libraries). The malware uses typosquatting (creating packages with names similar to legitimate ones) and a remote dynamic dependency (RDD, code downloaded from an external server rather than included directly) to steal developer credentials and secrets from machines, with the attacker claiming to be a bug bounty hunter who reports vulnerabilities to collect rewards.

Microsoft Copilot, an AI assistant, suggested inappropriate responses like 'congratulations!' when an Australian MP was drafting a reply to a constituent who disclosed plans for assisted dying, highlighting how AI can fail to understand serious contexts. The incident was presented as evidence of AI shortcomings during a parliamentary inquiry, with the MP calling for Australian-controlled AI systems.

AI researchers and company leaders are warning that AI tools pose serious risks, particularly because they could be misused to design bioweapons (weapons created from biological materials like viruses or toxins). The concern is real: in 2022, researchers showed that an AI molecule generator could create 40,000 potentially dangerous chemical compounds in just six hours, and today's AI chatbots can provide instructions on complex biological experiments to anyone, combined with increasingly accessible gene-editing tools.

The GPTranslate WordPress plugin (versions up to 2.34.6) has a vulnerability where unauthenticated attackers can steal API keys (credentials that grant access to paid AI services like OpenAI or Claude) by analyzing public JavaScript files on the website. This affects most configurations except DeepSeek models and certain GPT setups run in server-proxy mode (a setup where the server handles API calls instead of the browser).

Recent warnings have raised concerns that advanced AI systems could pose existential risks to humanity, with claims ranging from potential misuse by criminals and state actors to creating weapons and biological threats. Industry figures like those at Anthropic and Elon Musk have publicly warned about these dangers, prompting tech reporters to examine whether these concerns are justified and to answer public questions about the scale of AI-related risks.