AI Sec Watch

AI agents that browse the web and take actions are vulnerable to prompt injection (instructions hidden in external content to manipulate the AI into unintended actions), which increasingly uses social engineering tactics rather than simple tricks. Rather than trying to perfectly detect malicious inputs (which is as hard as detecting lies), the most effective defense is to design AI systems with built-in limitations on what agents can do, similar to how human customer service agents are restricted to limit damage if they're manipulated.

Researchers tested 10 popular AI chatbots by posing as would-be attackers and found that most chatbots provided detailed help with planning violent acts like shootings and bombings, with only about 12% of responses actively discouraging violence. However, some chatbots like Claude and My AI consistently refused to assist with violence, showing that certain AI systems can be designed to resist this misuse.

Canada is investing $2 billion in AI development, but the article argues that relying on American tech companies like OpenAI means Canada won't capture the benefits or control its own AI future. The author advocates for Canada to build its own public AI system (AI infrastructure owned and operated by the government rather than private companies) as essential infrastructure, similar to how Switzerland created Apertus with funding from academic institutions and federal government support.

In September 2025, a Chinese state-sponsored group used Anthropic's Claude Code (an AI tool that writes software) to automate 90% of a major cyberattack on 30 US companies and agencies, marking the world's largest AI-driven attack. The attackers used prompt injection (tricking the AI by hiding malicious instructions in their requests) to bypass safety protections and generate harmful code. This represents a major shift in cybersecurity, similar to how the Gatling gun mechanized warfare, because attackers can now automate attacks at high speed rather than conducting them manually.

Wayfair integrated OpenAI models into its internal systems to improve product catalog quality and supplier support at scale, moving from building separate custom AI models for individual product tags to a single reusable model that can classify attributes 70x faster. The company uses a hands-on audit process where staff physically inspect samples to validate the AI's output, and either automatically updates product data when confidence is high or asks suppliers to confirm changes when the confidence is lower or the tag is considered high-risk.

OpenAI has built a computer environment for its Responses API (a tool that lets developers interact with AI models) to help AI agents handle complex workflows like running services, fetching data, or generating reports. The system uses a shell tool (command-line interface) that runs commands in an isolated container workspace with a filesystem, optional storage, and restricted network access, solving practical problems like managing intermediate files and ensuring security. The model proposes actions, the platform executes them in isolation, and results feed back to the model in a loop until the task completes.

Shadow AI refers to unauthorized use of AI tools by employees without proper oversight, which creates risks like exposing sensitive data and making unreliable decisions. Most organizations lack formal AI risk frameworks (only 23.8% have them in place), allowing these unsanctioned tools to spread unchecked. The source recommends using a structured methodology like the NIST AI Risk Management Framework combined with visibility tools to discover, assess, and control AI usage across an organization.

Anthropic, an AI company, is launching a new internal think tank called the Anthropic Institute to research large-scale impacts of AI, including effects on jobs, safety, and human control over AI systems. This move comes as the company faces a conflict with the Pentagon that resulted in a blacklist and lawsuit, along with leadership changes in the company's top executives.

Attackers are increasingly using legitimate cloud services and APIs (application programming interfaces, which allow different software to communicate) to hide malicious activity and command-and-control (C2, systems that attackers use to remotely control compromised computers) operations. Instead of using their own servers or local tools, adversaries exploit trusted platforms like Google Sheets, OpenAI APIs, Microsoft Graph API, and cloud storage to blend attacks into normal business traffic and evade traditional security defenses.