AI Sec Watch

A story claimed that ChatGPT helped cure an Australian entrepreneur's dog of cancer, generating widespread attention as evidence that AI could revolutionize medicine. However, the article suggests this narrative is more complicated than the promoted version, implying the reality behind the claim differs from what was publicly reported.

The h3 library has a vulnerability in its Server-Sent Events (SSE, a protocol for pushing real-time messages from a server to connected clients) implementation where newline characters in message fields are not removed before being sent. An attacker who controls any message field (id, event, data, or comment) can inject newline characters to break the SSE format and trick clients into receiving fake events, potentially forcing aggressive reconnections or manipulating which past events are replayed.

Researchers discovered three connected flaws in Claude (an AI assistant) that can work together to steal user data, starting with a prompt injection attack (tricking the AI by hiding malicious instructions in its input) combined with a Google search vulnerability. This attack chain could potentially compromise enterprise networks that rely on Claude.

Shadow AI refers to AI systems hidden within SaaS applications (software services accessed online) that operate without proper oversight, creating security risks that can lead to major data breaches. The article emphasizes that organizations lack visibility into these autonomous AI systems and calls for better monitoring and control mechanisms to manage agentic AI (AI that can independently take actions to achieve goals).

Diffusion models (AI systems that generate images and other content by gradually removing noise from random data) are vulnerable to backdoor attacks, where hidden triggers cause the model to produce harmful outputs. Researchers created PureDiffusion, a framework that can both defend against these attacks by detecting and inverting the hidden triggers, and amplify attacks by making existing backdoors more effective.

This paper addresses backdoor attacks (where attackers secretly poison AI models to make them behave maliciously) in vertical federated learning (VFL, a setup where different organizations train an AI together on their own private data). The researchers propose a defense using a latent masked autoencoder (LMAE, a type of neural network that detects patterns and missing information) to identify when one participant is submitting suspicious, inconsistent data compared to honest participants, allowing the system to reject malicious contributions.

The @aborruso/ckan-mcp-server tool allows attackers to make HTTP requests to any address by controlling the `base_url` parameter, which has no validation or filtering. An attacker can use prompt injection (tricking the AI by hiding instructions in its input) to make the tool scan internal networks or steal cloud credentials, but exploitation requires the victim's AI assistant to have this server connected.

Langflow has a security flaw called IDOR (insecure direct object reference, where an attacker can access or modify resources belonging to other users) in its API key deletion feature. An authenticated attacker can delete other users' API keys by guessing their IDs, because the deletion endpoint doesn't verify that the API key belongs to the person making the request. This could allow attackers to disable other users' integrations or take over their accounts.

The Pentagon is planning to create secure environments where AI companies can train their models on classified military data, which would embed sensitive intelligence like surveillance reports into the AI systems themselves and bring these companies closer to classified information than before. This represents a major shift from current use of AI models like Claude in classified settings, but introduces unique security risks by allowing models to learn from rather than just access classified data.