AI Sec Watch

Researchers discovered that AI judges (LLMs acting as automated security gatekeepers to enforce safety policies) can be manipulated through prompt injection (tricking an AI by hiding instructions in its input) using stealthy formatting symbols rather than obvious gibberish. They created a tool called AdvJudge-Zero, a fuzzer (software that finds vulnerabilities by testing with unexpected inputs), which automatically identifies innocent-looking character sequences that exploit the model's decision-making logic to bypass security controls.

ChatGPT has introduced new interactive visual explanations for over 70 math and science concepts, allowing learners to manipulate variables and see real-time effects on graphs and outcomes instead of just reading static explanations. Research suggests that this type of interactive, visual learning helps students build stronger conceptual understanding compared to traditional instruction. The feature is now available globally to all ChatGPT users across all plans.

OpenAI is acquiring Promptfoo, a company that builds testing tools for AI applications, to improve security checks for AI agents (autonomous systems that operate independently in business processes) as more companies deploy them in production. Promptfoo's tools test AI models against adversarial prompts (malicious inputs designed to trick the AI), including prompt injection (hiding instructions in user input to manipulate the AI) and jailbreak attempts, and check whether models follow safety guidelines. The acquisition reflects growing enterprise concern about AI vulnerabilities and a shift toward treating AI security testing as an essential part of AI development, similar to traditional application security practices.

Katya, a freelance journalist turned content marketer, was recruited by Mercor to create training data for AI models by writing chatbot prompts and responses, work she initially enjoyed but which was abruptly canceled without warning. The article describes how machine learning (AI systems that improve by finding patterns in large amounts of data) relies on thousands of humans hired to generate and grade training examples, but gig workers like Katya face sudden project cancellations and job instability in this emerging industry.

Nvidia is planning to launch NemoClaw, an open-source platform for AI agents (specialized AI tools that can reason, plan, and act independently on complex tasks) targeting enterprise companies like Salesforce and Google. The platform will allow these companies to deploy AI agents to perform work tasks and is expected to include security and privacy tools, with early access offered to partners who contribute to the project.

Enterprise AI systems deployed for security work are heavily restricted by safety guardrails (automated filters designed to prevent harmful outputs), while attackers freely use jailbroken models (AI systems with safety measures bypassed), open-source alternatives, and purpose-built malicious tools. This creates an asymmetry where defenders face routine refusals when requesting legitimate defensive content like phishing simulations or proof-of-concept code, while attackers can easily circumvent safety measures through prompt injection (tricking AI by hiding instructions in its input) and other well-documented techniques, giving them a significant operational advantage.

Overseas 'content farms' based in Vietnam are using AI to create fake videos and images of UK politicians, spreading them on Facebook to go viral and potentially earn money through the platform's monetization program. The fake content, called deepfakes (digitally altered videos, pictures, or audio made to look real), depicts politicians in false situations like hospital stays or compromising scenarios, and Meta has removed some pages after investigation, though new ones continue appearing daily.

As generative AI (systems that create new content based on patterns in training data) becomes widespread across industries, organizations need specialized security tools to protect their AI infrastructure and data from cyber threats. AI Security Posture Management (AI-SPM) is a new category of security software designed to monitor, assess, and secure AI systems, complementing existing tools like CSPM (Cloud Security Posture Management, which protects cloud environments) and DSPM (Data Security Posture Management, which prevents data breaches).

More than 30 employees from OpenAI and Google DeepMind filed a court statement supporting Anthropic in a lawsuit against the U.S. Defense Department, which labeled the AI company a supply-chain risk after Anthropic refused to let the Pentagon use its technology for mass surveillance or autonomous weapons. The employees argue that the Pentagon could have simply canceled its contract with Anthropic and purchased from another AI company instead of designating it as a supply-chain risk, a label typically reserved for foreign adversaries. They contend that if the government is allowed to punish Anthropic this way, it will harm U.S. competitiveness in AI and discourage open discussion about the risks of AI systems.