HarnessEval-W: Agentifying the Evaluation of Visual Worlds
Overview
- Research Area: Computer Vision (CV)
- Authors: Weiliang Chen, Haowen Sun, Jun Gao et al. (43 authors)
- Release Date: 2026-08-17
- arXiv: 2608.16859
- Agentic evaluation for world models: First framework to apply the harness paradigm (familiar from LLM eval) to visual world model rollouts.
- Decomposable assessment: Evaluation questions are broken into measurable subproblems, each handled by a dedicated sub-agent.
- Diagnostic tool integration: Sub-agents are equipped with task-specific tools for inspecting physics, causality, and state evolution.
- Evidence tree outputs: Every verdict is accompanied by an auditable reasoning chain, not just a score.
- Open-source live benchmark: The entire pipeline is released for community evaluation and extension.
- arXiv link: <https://arxiv.org/abs/2608.16859>
Summary
A benchmark should deliver more than a scalar score: what makes an evaluation trustworthy is the reasoning that justifies the score. This is especially critical for world models, where judging a rollout requires understanding whether physics, causality, and world state evolve correctly. Humans spot such violations naturally, yet no existing benchmark automates this capability: metrics are computed brute-force, leaving no reasoning chain that can be examined or verified.
The authors introduce HarnessEval-W, an agentified evaluation pipeline that brings the harness paradigm from the LLM ecosystem to world model benchmarking. Rather than applying a fixed rubric, HarnessEval-W:
1. Interprets the context of each evaluation case, 2. Decomposes the evaluation question into measurable subproblems, 3. Spawns specialized sub-agents, each equipped with custom context and diagnostic tools to reason about its own subproblem. 4. A parent agent then validates the collected evidence and synthesizes a final verdict.
This hierarchical workflow transforms every evaluation into a transparent evidence tree, whose complete reasoning chain justifies the outcome. The full pipeline is open-sourced as a live benchmark, enabling community use and auditability.