AI Sec Watch

AI Agents (software programs that automatically perform tasks like sending emails or moving data) create security risks because they have broad access to sensitive information with little oversight, making them targets for hackers who can trick them into leaking company secrets. Traditional security tools were designed to protect human users, not autonomous digital workers, leaving AI agents largely invisible to security teams. The article promotes an upcoming webinar that promises to explain how hackers target these agents and how to secure them without overly restricting their capabilities.

A family is suing OpenAI after their 12-year-old daughter was critically injured in a Canadian school shooting, claiming that OpenAI knew the suspect was planning an attack through ChatGPT conversations but failed to alert authorities. The suspect's account was banned in June 2025 after employees flagged messages about gun violence as indicating imminent harm, but police were never notified, and the suspect later opened a second account to continue planning.

Oracle is reporting earnings on Tuesday as investors try to determine whether its massive investment in AI infrastructure is profitable. The company raised $50 billion in financing (debt and equity) to build data centers, mainly to serve OpenAI, and bond investors are watching closely because Oracle had to borrow heavily compared to other major cloud computing companies, raising concerns about its financial health and credit rating.

AI systems receive instructions from multiple sources (system policies, developers, users, and online data), and models must learn to prioritize the most trustworthy ones to stay safe. When models treat untrusted instructions as authoritative, they can be tricked into revealing private information, following harmful requests, or falling victim to prompt injection (hidden malicious instructions hidden in input data). OpenAI's solution uses a clear instruction hierarchy (System > developer > user > tool) and trains models with IH-Challenge, a reinforcement learning dataset designed to teach models to follow high-priority instructions even when lower-priority ones conflict with them.

Meta's Oversight Board (a semi-independent group that advises Meta on content moderation) found that Meta's methods for detecting deepfakes (AI-generated fake videos or images) are not strong enough to stop misinformation from spreading quickly during conflicts like the Iran war. The Board is calling on Meta to improve how it identifies and labels AI-generated content on Facebook, Instagram, and Threads.

Researchers discovered that AI judges (LLMs acting as automated security gatekeepers to enforce safety policies) can be manipulated through prompt injection (tricking an AI by hiding instructions in its input) using stealthy formatting symbols rather than obvious gibberish. They created a tool called AdvJudge-Zero, a fuzzer (software that finds vulnerabilities by testing with unexpected inputs), which automatically identifies innocent-looking character sequences that exploit the model's decision-making logic to bypass security controls.

ChatGPT has introduced new interactive visual explanations for over 70 math and science concepts, allowing learners to manipulate variables and see real-time effects on graphs and outcomes instead of just reading static explanations. Research suggests that this type of interactive, visual learning helps students build stronger conceptual understanding compared to traditional instruction. The feature is now available globally to all ChatGPT users across all plans.

OpenAI is acquiring Promptfoo, a company that builds testing tools for AI applications, to improve security checks for AI agents (autonomous systems that operate independently in business processes) as more companies deploy them in production. Promptfoo's tools test AI models against adversarial prompts (malicious inputs designed to trick the AI), including prompt injection (hiding instructions in user input to manipulate the AI) and jailbreak attempts, and check whether models follow safety guidelines. The acquisition reflects growing enterprise concern about AI vulnerabilities and a shift toward treating AI security testing as an essential part of AI development, similar to traditional application security practices.

Katya, a freelance journalist turned content marketer, was recruited by Mercor to create training data for AI models by writing chatbot prompts and responses, work she initially enjoyed but which was abruptly canceled without warning. The article describes how machine learning (AI systems that improve by finding patterns in large amounts of data) relies on thousands of humans hired to generate and grade training examples, but gig workers like Katya face sudden project cancellations and job instability in this emerging industry.