Paper Overview
Field: NLP Authors: Howard Zhang, Chuck Arvin, Logan Brown, Mihir Bhatia, Neelu Choudhary, Lily Miao Published: 2026-11-10 arXiv: 2611.14493
Abstract
Deploying LLM-based analytics agents in enterprise settings requires evaluation frameworks that can reliably detect failures across complex, multi-tool workflows. The authors present a three-phase comparative study of three evaluation frameworks, each representing a distinct evaluation paradigm (trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics), applied to two analytics agents in a controlled research setting using frozen execution traces.
- Phase 1 quantifies evaluation harness stability through repeated measurements, finding that 97% of configurations meet the stability threshold and that stability patterns are agent-independent.
- Phase 2 measures known risk detection on 72 ground-truth test cases, where the trace-based framework achieves the best precision-recall balance (F1 = 0.90). Text-only evaluation matches recall but produces 5x more false positives.
- Phase 3 evaluates unknown risk discovery, uncovering 24 novel failure types with zero overlap between functional testing and safety red-teaming.
Key Findings
1. The information available to an evaluation framework determines evaluation quality. 2. Functional and safety evaluations are complementary. 3. LLM-based evaluation can expose subtle errors that human review misses.
Recommendation
The authors recommend a dual-framework strategy: trace-based evaluation for functional assessment, combined with adversarial red-teaming for safety verification.
--- *Auto-collected on 2026-11-12*