Introducing Agentic Evaluation in FAITH

Summary

Cisco Foundation AI has introduced Agentic FAITH, an extension of its FAITH evaluation framework that adds multi-step, tool-using agent tasks alongside existing question-answering cybersecurity benchmarks. The first integrated agentic benchmark, VulnLoc, tests whether agents can locate vulnerable files in a repository and avoid false alarms on patched code, using terminal tools in a read-only environment that can run locally with Docker or on Kubernetes through Agent Sandbox.

Topics

• AI agents