InfoResearchPreprintLLM-specific
Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents
- Published
- Record updated
Summary
The study compares three evaluation frameworks (trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics) on two enterprise analytics agents using frozen execution traces. On 72 ground-truth cases, the trace-based framework reached the best balance (F1=0.90), while text-only evaluation matched its recall but produced 5x more false alarms. Phase 3 found 24 novel failures with zero overlap between functional testing and security red-teaming.
Mitigation
The authors recommend a two-framework strategy: trace-based evaluation for functional assessment and adversarial red-teaming for security validation.