Desktop-Delta Bench: A Step-Level Benchmark Testing Whether Computer-Use Models Understand Desktop GUI Transitions

Forum topic · 小凯 · 2026-07-30

Summary

Desktop-Delta Bench (DDB) is an offline, step-level benchmark that evaluates whether computer-use agents (CUAs) can reconstruct causal, task-relevant state transitions in desktop GUIs—capabilities that end-task success and single-frame grounding benchmarks fail to isolate. DDB contains 2,013 human-verified instances built from multi-app Linux trajectories spanning about 15 applications and 50 task domains, organized into two complementary tasks: 463 three-frame temporal ordering instances (including 105 cross-trajectory distractors) and 1,550 before/after action pairs annotated across five action-payload types. Evaluation of 8 open- and closed-source model families across 32 ordering and 16 single-action settings reveals consistent gaps. Ordering accuracy is far from saturated: best exact-match rates are 65.1% (non-distractor) and 65.7% (distractor). Task context improves distractor recognition by 6.9 points but reduces non-distractor exact match by 2.2 points, with models systematically copying the presented A-B-C order. Single-action results show inferring the action family is harder than grounding it: click F1 is 0.96 versus 0.76 for drag, though identified drags localize well. DDB fills the diagnostic gap between GUI grounding and end-task success, enabling targeted improvements in CUA verification, reliability, and recovery (arXiv: 2607.26041).

Paper Overview

Field: Computer Vision (CV) Authors: Abhishek Pillai, Samir Kumar Nayak, Yuan Chen Published: 2026-07-28 arXiv: 2607.26041

Full Abstract Translation

Computer-use agents (CUAs) increasingly act through desktop GUIs to complete long-horizon tasks. Current benchmarks primarily measure end-task success or single-frame grounding. Neither isolates whether a model can reconstruct the causal, task-relevant transition produced by an action—crucial for rejecting stale observations, verifying progress, and recovering from failure.

This is difficult because inference, remote input, app rendering, and screenshot capture are asynchronous: the next observation may be delayed, occluded, transient, or unrelated, then misread as progress and carried into subsequent planning.

The authors introduce Desktop-Delta Bench (DDB), an offline step-level benchmark with 2,013 human-verified instances from novel, multi-app Linux trajectories across ~15 applications and 50 task domains. DDB probes three failure dimensions—state verification, source tracking, and context-aware control—via two complementary tasks:

463 three-frame temporal ordering instances (including 105 cross-trajectory distractors)
1,550 annotated before/after action pairs spanning five action-payload types

Key Findings

Evaluation covers 8 closed- and open-source model families across 32 ordering and 16 single-action settings, revealing consistent gaps.
Ordering performance is far from saturated: best non-distractor and distractor exact-match rates are 65.1% and 65.7% respectively.
Task context improves distractor identification by 6.9 percentage points but reduces non-distractor exact match by 2.2 percentage points; error analysis shows systematic copying of the presented A-B-C order.
Single-action results show that inferring the action family is harder than localizing it: click F1 is 0.96 versus 0.76 for drag, though identified drags are generally well localized.

DDB complements end-to-end benchmarks by filling the missing diagnostic layer between GUI grounding and final task success, enabling targeted improvements in verification, reliability, and recovery for desktop CUAs.

--- *Auto-collected on 2026-07-30.*