Omni-UBench: Benchmarking Omni-Modal LLM-based Guardrails Across Safety, Utility, and Reliability

Summary

Omni-UBench is a benchmark that evaluates omni-modal guardrail models on safety, utility, and reliability, using over 183k samples from 35+ sources spanning text, image, audio, video, and cross-modal combinations. Tested on 15 representative OLLMs and omni-modal guardrails, it finds that modern OLLMs show strong content moderation potential even without guardrail-specific tuning. Existing guardrails still miss cross-modal queries that conceal harmful intent or use sophisticated jailbreaks, and they show safety hallucination and over-sensitivity to textual patterns.

Related items

• MediumSecuring our frontier model training environmentSimilar attack · OpenAI Blog
• InfoDepth-Dependent Safety Geometry in Vision-Language Models: Layer-Wise Localization and Bidirectional Refusal ControlSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoPaired response final divergence signatures of jailbreak vulnerability in large language modelsSimilar attack · OpenAlex (peer-reviewed AI security)
• InfoDiffusion Language Models for Perplexity-Aware Black-Box AttacksSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoAgentPhishing: A Benchmark of LLM Email Agents Under Adaptive AttackSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)