Desktop-Delta Bench: A Step-Level Benchmark for Computer-Use Model GUI Reasoning

Forum topic · 小凯 · 2026-07-30

Summary

Desktop-Delta Bench (DDB) is an offline, step-level benchmark (arXiv:2607.26041) designed to test whether computer-use agents (CUAs) can reconstruct the causal, task-relevant state transitions produced by their actions in desktop GUI environments. Unlike existing benchmarks that measure end-task success or single-frame grounding, DDB contains 2,013 human-verified instances drawn from multi-app Linux trajectories spanning roughly 15 applications and 50 task domains. It includes 463 three-frame temporal ordering instances (with 105 cross-trajectory distractors) and 1,550 before/after action pairs annotated across five action and payload types. Evaluations of eight closed- and open-source model families reveal consistent gaps: best exact-match accuracy is only about 65% on ordering tasks, and models systematically copy the presented A-B-C order. Action-family inference is harder than localization—dragging scores F1 0.76 versus 0.96 for clicks. Task context improves distractor rejection by 6.9 points but slightly hurts non-distractor accuracy. DDB fills a diagnostic layer between GUI grounding and end-task success, enabling targeted improvements in CUA verification, reliability, and recovery.

Paper Overview

Field: Computer Vision (CV)
Authors: Abhishek Pillai, Samir Kumar Nayak, Yuan Chen
Published: 2026-07-28
arXiv: 2607.26041

Abstract (Translation)

Computer-use agents (CUAs) increasingly act through desktop GUIs to complete long-horizon tasks. Current benchmarks primarily measure end-task success or single-frame grounding. Neither isolates whether a model can reconstruct the causal, task-relevant transition produced by an action—crucial for rejecting stale observations, verifying progress, and recovering from failure. This is difficult because inference, remote input, app rendering, and screenshot capture are asynchronous: the next observation may be delayed, occluded, transient, or unrelated, then misread as progress and carried into subsequent planning.

The authors introduce Desktop-Delta Bench (DDB), an offline step-level benchmark with 2,013 human-verified instances from novel, multi-app Linux trajectories across ~15 applications and 50 task domains. DDB trajectories probe three failure dimensions—state verification, source tracking, and context-aware control—through two complementary tasks:

463 three-frame temporal ordering instances, including 105 cross-trajectory distractors
1,550 annotated before/after pairs spanning 5 action and payload types

Key Findings

Eight closed- and open-source model families were evaluated across 32 ordering and 16 single-action settings, revealing consistent gaps.
Ordering performance is far from saturated: best exact-match accuracy is 65.1% (non-distractor) and 65.7% (distractor).
Task context improves distractor identification by 6.9 points but reduces non-distractor exact-match by 2.2 points; error analysis shows systematic copying of the presented A-B-C order.
Inferring the action family is harder than localizing it: click F1 is 0.96 versus 0.76 for drag, though identified drags are usually well localized.

Significance

DDB fills a missing diagnostic layer between GUI grounding and end-task success, complementing end-to-end benchmarks and enabling targeted improvements in verification, reliability, and recovery for desktop CUAs.

--- *Auto-collected 2026-07-30*