AI Sec Watch

AI models frequently make errors or hallucinate (generate false or inaccurate information) when recommending which software versions to use, how to upgrade systems, or which security fixes to apply, which can create significant technical debt (accumulated costs from shortcuts and poor decisions that must eventually be addressed). These mistakes can lead developers to ignore real security bugs or choose problematic upgrade paths.

Conntour is an AI-powered video search platform that uses vision-language models (AI systems trained to understand both images and text) to let security personnel search through surveillance footage using natural language queries, similar to how Google searches the web. The startup raised $7 million in funding and distinguishes itself by efficiently scaling to handle thousands of camera feeds while running on standard consumer hardware like Nvidia GPUs. The company's founders emphasize being selective about which clients they work with based on ethical and legal considerations.

Graph Neural Networks (GNNs, AI systems designed to work with interconnected data structured as graphs) used in graph self-supervised learning (training without labeled data) can be secretly compromised by backdoor attacks (where hidden malicious instructions are embedded in the model). Researchers developed GDetox, a defense method that removes these backdoor features from compromised encoders (the parts of the model that learn to represent data) using knowledge distillation (a technique where a teacher model teaches a student model to learn better), reducing successful attacks to 4% while keeping the model's normal performance nearly unchanged.

Deepfake technology can create fake facial images that are hard to distinguish from real ones, posing risks to privacy and security. This paper proposes a new detection method using Visual Language Models (VLMs, AI systems that understand both images and text) combined with component-specific prompt tuning (customizing input instructions to focus on specific facial parts like eyes and nose). The approach transforms deepfake detection into a Visual Question Answering task and uses a Q-Former module (a feature extraction component guided by instructions) to help the model identify forgery traces in local facial features, achieving better accuracy than existing methods.

A vulnerability called ShadowPrompt in Anthropic's Claude Chrome extension allowed attackers to inject malicious prompts (hidden instructions) into the AI without user interaction by exploiting two flaws: an overly permissive allowlist that trusted any subdomain matching *.claude.ai, and an XSS vulnerability (a security flaw allowing attackers to run malicious code) in an Arkose Labs CAPTCHA component. This zero-click attack could let attackers steal sensitive data, read conversation history, or perform actions like sending emails on behalf of the victim.

European lawmakers voted to delay compliance deadlines for the EU AI Act, pushing back requirements for developers of high-risk AI systems (those that could seriously harm health, safety, or people's rights) until December 2027, with even later deadlines for AI used in regulated sectors like medical devices. The Parliament also backed proposals to ban nudify apps, which use AI to create fake nude images of people without consent.

Eline van der Velden created an AI actor called Tilly Norwood (a digital twin, or an AI-generated copy of a person) and received death threats following global backlash against the project. Van der Velden stated she developed it to spark discussion about AI's impact on entertainment, but the reaction from Hollywood actors and unions was more severe than expected.

OpenAI has indefinitely paused plans to release an 'adult mode' for ChatGPT, a sexualized chatbot feature that faced criticism from employees and investors over potential harms to society. This decision is part of a broader company refocus on core products, following similar discontinuations like the text-to-video platform Sora.

The Trump administration issued an executive order that prevents states from regulating AI by threatening to sue them and cut their funding, which supports tech industry interests but goes against what voters want. Polls show over 70% of voters favor state and federal regulation of AI, yet the administration sided with industry lobbyists instead, creating a major political divide ahead of midterm elections. Local communities across the country are already resisting AI datacenters due to environmental and energy concerns, with both progressive and Trump-supporting voters working together against the development.