AI Sec Watch

OpenAI is shifting its research focus toward building an AI researcher, a fully automated agent-based system (software that can act independently to complete tasks) capable of tackling complex problems in math, physics, biology, and other fields without human intervention. The company plans to release an autonomous AI research intern by September 2024, with a more advanced multi-agent system (multiple AI agents working together) by 2028. OpenAI's chief scientist says the goal is to create systems that can work for extended periods with minimal human guidance, eventually enabling "a whole research lab in a data center."

PinchTab is an HTTP server (a program that handles web requests) that lets AI agents control a Chrome web browser. Versions 0.8.2 and earlier have a blind SSRF vulnerability (a flaw where an attacker tricks the server into making requests to internal networks that should be off-limits) in the /download endpoint, because the server only checks the URL once but the browser can follow hidden redirects to reach internal addresses. The risk is limited because the vulnerable feature is disabled by default.

A survey by Anthropic of about 81,000 people across 159 countries found that people in Sub-Saharan Africa and Asia are more optimistic about AI than those in Western Europe and North America, with most respondents hoping AI will help them earn money and be more productive at work. However, independent workers like entrepreneurs have benefited far more from AI than salaried employees, and concerns about job displacement affect about 22% of respondents as agentic AI (AI systems that can perform complex tasks with minimal human direction) becomes more capable.

Cybercriminals are using AI to launch more effective attacks, including personalized phishing emails, deepfakes, and malware that mimics normal user behavior to evade traditional security tools. Traditional detection methods like signature-based detection (identifying threats by their known code patterns) and rule-based systems (using preset thresholds for suspicious activity) fail against these AI-enabled attacks because the malware constantly changes and the criminal behavior blends in with legitimate activity. The source emphasizes that organizations need to shift from rule-based monitoring to behavioral analytics using dynamic, identity-based risk modeling that can detect inconsistencies in real time.

FastGPT (an AI platform for building AI agents) versions 4.14.8.3 and below have a critical security flaw where the fastgpt-preview-image.yml workflow uses pull_request_target (a GitHub feature that runs code with access to repository secrets) but executes code from an external contributor's fork, allowing attackers to run arbitrary code (commands on systems they don't own), steal secrets, and potentially compromise the production container registry (the central storage system for packaged software).

A Meta employee asked an AI agent for help with an engineering problem on an internal forum, and the AI's suggested solution caused a large amount of sensitive user and company data to be exposed to engineers for two hours. This incident demonstrates a risk where AI systems can inadvertently guide people toward actions that create security problems, even when the person following the guidance has good intentions.

SQLBot, an intelligent data query system that uses a large language model and RAG (retrieval-augmented generation, where an AI pulls in external documents to answer questions), has a critical SQL injection vulnerability (a bug where an attacker tricks the system into running unintended database commands) in versions before 1.7.0 that allows authenticated users to execute arbitrary code on the backend server. The vulnerability exists because Excel sheet names are directly inserted into database commands without proper sanitization (cleaning/validation), and attackers can exploit this by uploading specially crafted files to gain complete control of the system.

SQLBot, an AI-based system for querying databases that uses RAG (retrieval-augmented generation, where it pulls in external data to answer questions), has a vulnerability in versions before 1.7.0 that lets attackers read any file from the server. An attacker can exploit the /api/v1/datasource/check endpoint by submitting a fake MySQL connection with a malicious setting, which tricks the server into reading and sending back sensitive files like /etc/passwd when it tries to verify the connection.

OpenAI is combining its web browser, ChatGPT app, and Codex app (a tool for writing and understanding code) into a single desktop application to simplify the user experience and reduce fragmentation across its products. The company is refocusing its efforts on high-productivity use cases and avoiding distractions as it prepares for a potential IPO.