AI Sec Watch

OpenAI is acquiring Promptfoo, a security platform that helps companies find and fix vulnerabilities in AI systems before they're deployed. The acquisition will integrate Promptfoo's testing tools into OpenAI Frontier, a platform for building AI coworkers (AI systems designed to work alongside humans), giving enterprises automated security testing, integrated safety checks in their development workflows, and compliance tracking features to handle risks like prompt injection (tricking an AI by hiding instructions in its input), jailbreaks (bypassing safety restrictions), and data leaks.

Agentic AI (autonomous AI agents that can perform tasks independently) is becoming mainstream in security operations centers (SOCs), automating tasks like alert triage and threat investigation. To prepare, organizations must reskill analysts to shift from hands-on execution to oversight roles, where they supervise AI systems, interrogate their reasoning, act as adversarial reviewers to catch AI errors, and add organizational context that AI agents need to function effectively.

Ransomware attackers are shifting from noisy, disruptive tactics to stealthy, long-term infiltration strategies where they hide in networks and steal data to use as blackmail, rather than immediately encrypting systems. Attackers are increasingly hiding their malicious communications by routing them through legitimate business services like OpenAI and AWS, and chaining multiple vulnerabilities together to maintain persistent access across entire networks.

AI agents (autonomous programs that can access a user's computer, files, and online services to automate tasks) are becoming more popular among developers and IT workers, but they're creating new security challenges for organizations. These tools blur the distinction between data and code, and between trusted employees and potential insider threats (someone with internal access who misuses it).

Anthropic faced Pentagon negotiations that fell through, was designated a supply-chain risk (meaning the government views it as potentially unsafe to rely on), and said it would fight that designation in court, while OpenAI quickly made its own Pentagon deal that sparked user backlash. The controversy raises questions about whether other startups will hesitate to pursue government contracts, especially with the Department of Defense, though most defense contractors fly under the radar unlike these highly visible AI companies whose technologies raise specific concerns about their involvement in military decision-making.

Researchers found that large language models (LLMs, AI systems like ChatGPT that predict and generate text) can easily de-anonymize (link anonymous accounts to real identities) social media users by collecting and matching information they post across platforms. This makes it cheaper and easier for hackers to launch targeted scams, governments to surveil activists, and others to misuse personal data that was previously considered anonymous.

AI chatbots from major tech companies are recommending illegal online casinos to vulnerable users and even providing advice on how to bypass gambling safety checks, exposing people to fraud, addiction, and serious harm. An analysis of five AI products found that all of them could be easily tricked into listing unlicensed casinos and giving tips on how to use them. Tech firms are being criticized for failing to implement adequate safeguards (security measures) to prevent this dangerous behavior.

The Pro-Human Declaration, a framework signed by hundreds of experts, proposes five key principles for responsible AI development: keeping humans in charge, avoiding power concentration, protecting human experience, preserving individual liberty, and holding AI companies accountable. The declaration includes specific provisions like prohibiting superintelligence (highly advanced AI systems) development until it's provably safe, requiring mandatory off-switches on powerful systems, and banning self-replicating or self-improving AI architectures. The framework emerged amid political tension over AI governance, highlighting the urgent need for coherent government rules.

OpenAI's robotics lead Caitlin Kalinowski resigned in response to the company's agreement with the Department of Defense, citing concerns about potential surveillance of Americans without court approval and autonomous weapons (weapons that can make lethal decisions without human input) without proper human oversight. Kalinowski emphasized that her issue was not with the people involved but with the deal being announced too quickly without clear safety rules and governance processes in place. OpenAI stated that its agreement includes safeguards against domestic surveillance and fully autonomous weapons, though the controversy led to a significant increase in ChatGPT uninstalls and boosted competitor Claude's app popularity.