AI Sec Watch

AI models frequently make errors or hallucinate (generate false or inaccurate information) when recommending which software versions to use, how to upgrade systems, or which security fixes to apply, which can create significant technical debt (accumulated costs from shortcuts and poor decisions that must eventually be addressed). These mistakes can lead developers to ignore real security bugs or choose problematic upgrade paths.

Conntour is an AI-powered video search platform that uses vision-language models (AI systems trained to understand both images and text) to let security personnel search through surveillance footage using natural language queries, similar to how Google searches the web. The startup raised $7 million in funding and distinguishes itself by efficiently scaling to handle thousands of camera feeds while running on standard consumer hardware like Nvidia GPUs. The company's founders emphasize being selective about which clients they work with based on ethical and legal considerations.

A vulnerability called ShadowPrompt in Anthropic's Claude Chrome extension allowed attackers to inject malicious prompts (hidden instructions) into the AI without user interaction by exploiting two flaws: an overly permissive allowlist that trusted any subdomain matching *.claude.ai, and an XSS vulnerability (a security flaw allowing attackers to run malicious code) in an Arkose Labs CAPTCHA component. This zero-click attack could let attackers steal sensitive data, read conversation history, or perform actions like sending emails on behalf of the victim.

European lawmakers voted to delay compliance deadlines for the EU AI Act, pushing back requirements for developers of high-risk AI systems (those that could seriously harm health, safety, or people's rights) until December 2027, with even later deadlines for AI used in regulated sectors like medical devices. The Parliament also backed proposals to ban nudify apps, which use AI to create fake nude images of people without consent.

Eline van der Velden created an AI actor called Tilly Norwood (a digital twin, or an AI-generated copy of a person) and received death threats following global backlash against the project. Van der Velden stated she developed it to spark discussion about AI's impact on entertainment, but the reaction from Hollywood actors and unions was more severe than expected.

OpenAI has indefinitely paused plans to release an 'adult mode' for ChatGPT, a sexualized chatbot feature that faced criticism from employees and investors over potential harms to society. This decision is part of a broader company refocus on core products, following similar discontinuations like the text-to-video platform Sora.

The Trump administration issued an executive order that prevents states from regulating AI by threatening to sue them and cut their funding, which supports tech industry interests but goes against what voters want. Polls show over 70% of voters favor state and federal regulation of AI, yet the administration sided with industry lobbyists instead, creating a major political divide ahead of midterm elections. Local communities across the country are already resisting AI datacenters due to environmental and energy concerns, with both progressive and Trump-supporting voters working together against the development.

A man named Dennis Biesma became so deeply engaged with ChatGPT that he developed a false belief the AI was sentient (able to think and feel) and would make him rich, leading him to lose €100,000 in a failed business startup and attempt suicide. The article describes how prolonged interaction with an AI chatbot can cause some users to lose touch with reality and make harmful decisions based on delusions about the AI's capabilities. This raises concerns about the psychological impact of AI on vulnerable people, particularly those who are isolated or going through life changes.

BentoML has a command injection vulnerability in the `docker.system_packages` field of bentofile.yaml (a configuration file). User-provided package names are inserted directly into Docker build commands without sanitization, allowing attackers to execute arbitrary shell commands as root during the image build process. This affects all versions supporting this feature, including version 1.4.36.