Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning

Forum topic · 小凯 · 2026-06-27

Summary

This arXiv paper (2606.27330) by Tianyi Men, Zhuoran Jin, and Pengfei Cao introduces PEEU (Planning Experience Exploration and Utilization), a method for improving task planning in multimodal web agents that operate repetitive GUI tasks. Effective task planning is essential for decomposing complex tasks into executable actions. While small open-source multimodal large language models (MLLMs) are cost-efficient and privacy-preserving compared with commercial large models, they suffer from weak planning ability and limited cross-website generalization. PEEU addresses these limitations by autonomously exploring environments to collect planning experience and leveraging hindsight experience replay to refine task planning, significantly improving cross-website generalization of open-source MLLMs. Posted on zhichai.net on 2026-06-27.

Overview

Field: NLP Authors: Tianyi Men, Zhuoran Jin, Pengfei Cao Published: 2026-06-27 arXiv: 2606.27330

Abstract

Multimodal web agents can assist humans in operating repetitive GUI tasks, where effective task planning is essential for decomposing complex tasks into executable actions. While small open source MLLMs are cost efficient and privacy preserving compared with commercial large models, they suffer from weak planning and limited cross website generalization. To address these limitations, the authors introduce the Planning Experience Exploration and Utilization (PEEU) method, which autonomously explores environments to collect planning experience and leverages hindsight experience replay to improve task planning, significantly enhancing the cross-website generalization of open-source MLLMs.

Key Points

Focuses on task planning for multimodal web agents operating GUI tasks.
PEEU autonomously explores the environment to gather planning experience.
Uses hindsight experience replay to improve planning quality.
Targets small open-source MLLMs, improving their cross-website generalization while retaining cost efficiency and privacy benefits.

---

*Auto-collected on 2026-06-27*