Overview
Field: NLP Authors: Shiyu Xuan, Zechao Li arXiv: 2608.11191
Abstract
GUI Visual Grounding is a fundamental capability for GUI agents. Existing models typically freeze their parameters after deployment, limiting their ability to adapt to unseen interfaces. Although recent methods attempt to adapt models via test-time reinforcement learning, they cannot reflect upon failed exploration. To overcome this, the authors propose a Test-Time Self-Evolving framework that enables models to improve after deployment without human-annotated ground truth.
Method
The framework constructs a closed loop of Exploration, Evaluation, Reflection, and Internalization:
- Exploration: the agent explores unseen interfaces by predicting grounding coordinates for given instructions.
- Evaluation: an MLLM-based Reflector assesses the generated results and provides reasoning reflections.
- Internalization: reflection-guided on-policy self-distillation internalizes reflection knowledge into model weights, converting high-level reasoning into dense token-level supervision through a conditioned self-teacher.
- Contrastive calibration: a designed method prevents erroneous autoregressive prefixes from corrupting the supervision signal during failed explorations.
- Extensive experiments across six benchmarks demonstrate the framework's effectiveness, with an average accuracy improvement of 7.4% over the base model.
- According to the authors, this is the first work to successfully leverage on-policy self-distillation for test-time adaptation in GUI visual grounding.
- By filling the post-deployment adaptation gap, the framework completes the self-evolving capability of GUI agents.
Results
*Original paper: arXiv:2608.11191*