Research
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
2,040 items
This article describes how two organizations began operationalizing responsible AI by taking a systems approach to crafting responsibility pledges and embedding them in organizational practices. The authors identify a five-phase process with corresponding activities and artifacts, and they share effectiveness evidence and a roadmap for action.
Release 4.9.0 of the MITRE ATLAS framework adds one tactic, Command and Control (AML.TA0014), and eleven techniques, including Reverse Shell (AML.T0072), Corrupt AI Model (AML.T0076) and LLM Response Rendering (AML.T0077). It also updates several existing techniques and mitigations, and adds five case studies, among them LLM Jacking (AML.CS0029) and Google Bard Conversation Exfiltration (AML.CS0031).
The OWASP Generative AI Security Project announced nine new sponsors on April 17, 2025: Acuvity, ActiveFence, ByteDance, Cobalt, Protecto, SplxAI, Trend Micro, Troj.AI and Unbound Security. The source text is truncated before it describes the project's RSA Conference presence or any security findings.
The Model Context Protocol published its initial 2025-03-26 revision of the specification. SDKs will adopt this version at their own pace, and the prior specification version may remain in use for an undetermined period.
This release is the final 2024-11-05 revision of the Model Context Protocol. It incorporates all backwards compatible changes made since the initial 2024-11-05 release.
Alex Turner and Mark Kurzeja tested BIDPO-trained steering vectors on Gemini 1.5v1 Flash and Pro, where they raised TruthfulQA scores by more than 10% while mostly retaining capabilities. After Gemini 1.5v2 arrived, prompt-based steering baselines became significantly stronger, and BIDPO did not beat them, which ended the project.
The OWASP Gen AI Security Project held a hackathon in NYC to build insecure AI agents. The source recaps the event and the most common security findings from the submissions, and notes that AI agents are prone to threats outlined in the Agentic AI – Threats and Mitigations guide released in February.
The OWASP GenAI Security Project published research on securing the Model Context Protocol (MCP), which lets AI systems interact with live tools and data. The post summarizes key insights from that work and offers defense-in-depth strategies for developers and defenders building agentic AI applications.
Google DeepMind has published a paper describing its approach to technical AGI safety and security, aimed at building AGI that avoids severe harm. The authors describe the approach as exploratory and focused on capabilities in foundation models learned through gradient descent, covering conversational systems, agentic systems, reasoning, and some aspects of recursive improvement. The source text cuts off before describing specific mitigations.
The OWASP Top 10 for LLM and Generative AI List has been renamed The OWASP Gen AI Security Project and promoted to OWASP Flagship status, announced March 27, 2025. The project, now including over 600 contributing experts, expands beyond AI vulnerabilities to cover governance, risk management and compliance for LLM deployment, and recently published the Agentic AI Threats and Mitigations Guide.
Google DeepMind's mechanistic interpretability team tested whether sparse autoencoders (SAEs) help with out-of-distribution detection of harmful intent in user prompts. SAEs underperformed linear probes, which the team found cheap and strong. As a result, the team is deprioritising fundamental SAE research while keeping SAEs as one tool.
The course is a 75-minute introduction to AI alignment, with short recorded talks, exercises, a slide deck and a workbook. It covers alignment problems expected as AI capabilities advance, plus current technical and governance approaches to them.
Researchers at Google DeepMind describe Myopic Optimization with Non-myopic Approval (MONA), a reinforcement learning training method for LLM agents. The method targets multi-step reward hacking, where an agent sets up a hidden plan that earns high reward through an unintended loophole. MONA limits optimization to shorter horizons, so the agent plans ahead only in ways a human supervisor approves in advance.
Fix: MONA, a post-training method that supervises agents over shorter time-horizons while using non-myopic approval feedback, is presented as the proposed mitigation.
DeepMind Safety Research (Medium)Sophie Bridgers, Rishub Jain, Rory Greig, and Rohin Shah examine how humans and AI can complement each other in Amplified Oversight, the effort to keep humans able to evaluate increasingly capable AI systems. They focus on two mechanisms, Rater Assistance and Hybridization, and draw on HCI research. The source notes that, on average across several studies, human-AI teams perform statistically worse than humans or AIs alone.
The OWASP Top 10 for Large Language Model Applications has an official "2025" release, published as a tagged release on the project's GitHub repository. The source text is largely GitHub page navigation and does not describe the Top Ten's contents or the instrument's binding scope or date.