English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation

Forum topic · 小凯 · 2026-08-13

Summary

A new arXiv paper (2608.11191) by Shiyu Xuan and Zechao Li proposes a test-time self-evolving framework for GUI visual grounding, a core capability of GUI agents. Existing models freeze parameters after deployment and struggle on unseen interfaces, while prior test-time reinforcement learning methods cannot reflect on failed exploration. The proposed framework builds a closed loop of Exploration, Evaluation, Reflection, and Internalization: an agent predicts grounding coordinates on unseen interfaces, an MLLM-based Reflector assesses results and generates reasoning reflections, and reflection-guided on-policy self-distillation converts high-level reasoning into dense token-level supervision via a conditioned self-teacher. A contrastive calibration method prevents erroneous autoregressive prefixes from corrupting supervision signals during failed explorations. Experiments across six benchmarks show an average accuracy improvement of 7.4% over the base model without human-annotated ground truth. The authors state this is the first work to successfully apply on-policy self-distillation for test-time adaptation in GUI visual grounding, and code will be open-sourced.

Overview

Field: NLP Authors: Shiyu Xuan, Zechao Li arXiv: 2608.11191

Abstract

GUI Visual Grounding is a fundamental capability for GUI agents. Existing models typically freeze their parameters after deployment, limiting their ability to adapt to unseen interfaces. Although recent methods attempt to adapt models via test-time reinforcement learning, they cannot reflect upon failed exploration. To overcome this, the authors propose a Test-Time Self-Evolving framework that enables models to improve after deployment without human-annotated ground truth.

Method

The framework constructs a closed loop of Exploration, Evaluation, Reflection, and Internalization:

  • Exploration: the agent explores unseen interfaces by predicting grounding coordinates for given instructions.
  • Evaluation: an MLLM-based Reflector assesses the generated results and provides reasoning reflections.
  • Internalization: reflection-guided on-policy self-distillation internalizes reflection knowledge into model weights, converting high-level reasoning into dense token-level supervision through a conditioned self-teacher.
  • Contrastive calibration: a designed method prevents erroneous autoregressive prefixes from corrupting the supervision signal during failed explorations.
  • Results

  • Extensive experiments across six benchmarks demonstrate the framework's effectiveness, with an average accuracy improvement of 7.4% over the base model.
  • According to the authors, this is the first work to successfully leverage on-policy self-distillation for test-time adaptation in GUI visual grounding.
  • By filling the post-deployment adaptation gap, the framework completes the self-evolving capability of GUI agents.
Code will be open-sourced.

*Original paper: arXiv:2608.11191*

Tags

#gui-agents#visual-grounding#test-time-adaptation#self-distillation#multimodal-llm#reinforcement-learning#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633404