AI Sec Watch

Dario Amodei, CEO of Anthropic, has called for AI development to slow down and be closely monitored because the risks are "serious." He proposed a three-point plan including independent monitoring of AI models as they develop, industry-wide regulation, and global regulation, and committed Anthropic to building AI at a "balanced rate" that ensures safety while still advancing the technology. Other AI leaders like OpenAI's Sam Altman and Elon Musk have expressed support for slowing down AI development and using independent evaluators (third-party monitors who check if AI models are safe before release) to assess safety.

Anthropic CEO Dario Amodei published an essay proposing that AI companies voluntarily slow their development pace, citing concerns that current AI models are becoming powerful enough to pose safety risks. His three-step plan includes allowing third-party evaluators employee-level access to verify safety practices, encouraging leading AI companies to establish common safety standards, and coordinating between democratic and authoritarian governments, with Amodei emphasizing that pacing means taking time to align and safeguard models rather than halting development entirely.

Anthropic's CEO argues that AI companies should slow down development to allow time for safety measures and regulatory review. The company is voluntarily giving third-party evaluators (like METR, an independent AI safety organization) access to its models so they can check whether the company is following its safety commitments.

Anthropic's CEO Dario Amodei called for the AI industry to slow its development pace and proposed a three-part plan to do so. As part of this plan, Anthropic committed to giving third-party evaluators (independent outside experts) permanent access to their AI systems so these evaluators can check that safety measures are being followed, report problems, and assess how well the AI models behave during training.

vLLM (an open-source LLM serving framework) versions before 0.28.0 don't properly check audio sample rate headers (the metadata describing how many audio samples are recorded per second) in its transcription endpoint, letting authenticated users submit fake audio headers with inflated sample rates that cause the server to allocate excessive memory and crash. This affects all users of the affected vLLM instance.

vLLM versions 0.10.2 through 0.27.x have a vulnerability where audio extraction from video files lacks size and duration limits, allowing attackers to upload specially crafted compressed videos that force the server to use massive amounts of memory during decoding (a denial of service attack, where legitimate users can't access the service). This only affects NanoNemotronVL models when video audio processing is enabled.

vLLM (a tool for running large language models efficiently) versions before 0.28.0 have a remote code execution vulnerability (a flaw that lets attackers run their own code on a system) in the LlavaOnevision2 processor loader. Even when a user sets trust_remote_code to False (a safety setting meant to prevent loading untrusted code), the vulnerability ignores this setting and allows attackers to execute malicious code by hiding it in a model file.

Influencers are facing a new threat where deepfakes (AI-generated fake videos or images that look realistic) of them are being used in fake sponsored ads without their permission. Emily Schuman, a lifestyle influencer with over 500,000 followers, discovered multiple fake ads showing AI versions of herself promoting products like GLP-1 drugs, makeup, and blood tests, which confused her followers and damaged her credibility since she never actually endorsed these products.

OpenAI has recently claimed to solve a Millennium Prize Problem, a legendary unsolved mathematics challenge, marking a significant achievement in AI capabilities. However, many mathematicians are concerned about OpenAI's approach, viewing the company as a well-funded outsider that is aggressively pursuing these problems without respecting traditional academic norms or considering the impact on researchers who have devoted their careers to these fields.