Test-Time Self-Evolving Framework for GUI Visual Grounding

Forum topic · 小凯 · 2026-08-13

Summary

This paper introduces a Test-Time Self-Evolving framework that enables GUI visual grounding models to improve after deployment without human-annotated ground truth. GUI visual grounding is a fundamental capability for GUI agents, but existing models typically freeze their parameters post-deployment, limiting adaptability to unseen interfaces. While recent test-time reinforcement learning approaches attempt adaptation, they cannot reflect upon failed exploration. The proposed framework constructs a closed loop of Exploration, Evaluation, Reflection, and Internalization. An agent first explores unseen interfaces by predicting grounding coordinates for given instructions. An MLLM-based Reflector then assesses the predictions and provides reasoning feedback. To internalize this reflection knowledge into model weights, the authors propose reflection-guided on-policy self-distillation, which converts high-level reasoning into dense token-level supervision via a conditioned self-teacher. A contrastive calibration method prevents erroneous autoregressive prefixes from corrupting supervision during failed exploration. Experiments across six benchmarks demonstrate an average accuracy improvement of 7.4% over the base model. The work is reportedly the first to successfully leverage on-policy self-distillation for test-time adaptation in GUI visual grounding.

Paper Overview

Research Area: NLP
Authors: Shiyu Xuan, Zechao Li
Published: 2026-08-11
arXiv: 2608.11191

Abstract

GUI Visual Grounding is a fundamental capability for GUI agents. Existing models typically freeze their parameters after deployment, limiting their ability to adapt to unseen interfaces. Although recent methods attempt to adapt models via test-time reinforcement learning, they cannot reflect upon failed exploration.

To overcome this, we propose a Test-Time Self-Evolving framework that enables models to improve after deployment without human-annotated ground truth. It constructs a closed-loop of Exploration, Evaluation, Reflection, and Internalization.

Specifically, the agent first explores unseen interfaces by predicting grounding coordinates for given instructions. To evaluate these explorations, we introduce an MLLM-based Reflector to assess the generated results and provide the corresponding reasoning reflections. To internalize reflection knowledge into model weights, we propose reflection-guided on-policy self-distillation, converting high-level reasoning into dense token-level supervision via a conditioned self-teacher. In addition, we design a contrastive calibration method to prevent erroneous autoregressive prefixes from corrupting the supervision signal during failed exploration.

Extensive experiments across six benchmarks demonstrate the effectiveness of our framework, yielding an average accuracy improvement of 7.4% over the base model. To our knowledge, this is the first work to successfully leverage on-policy self-distillation for test-time adaptation in GUI visual grounding. By filling the post-deployment adaptation gap, our framework completes the self-evolving capabilities of GUI agents. The code will be open-sourced.

Key Points

Problem: Deployed GUI visual grounding models are frozen and cannot adapt to novel interfaces; existing test-time RL methods lack reflection over failed attempts.
Framework: A closed-loop pipeline of Exploration → Evaluation → Reflection → Internalization, applied at inference time without ground-truth labels.
Reflector: An MLLM judges predicted grounding coordinates and produces reasoning feedback.
Self-Distillation: Reflection-guided on-policy self-distillation uses a conditioned self-teacher to transform high-level reasoning into dense token-level supervision.
Robustness: Contrastive calibration avoids corruption of the supervision signal by erroneous autoregressive prefixes during failures.
Results: Average accuracy gain of 7.4% across six benchmarks over the base model.
Contribution claim: First successful use of on-policy self-distillation for test-time adaptation in GUI visual grounding.
Reference: arXiv:2608.11191