Paper Overview
Research Area: NLP Authors: Tianyi Men, Zhuoran Jin, Pengfei Cao Published: 2026-06-27 arXiv: 2606.27330
Abstract (Translated)
Multimodal web agents can assist humans in operating repetitive GUI tasks, where effective task planning is essential for decomposing complex tasks into executable actions. While small open-source MLLMs are cost-efficient and privacy-preserving compared with commercial large models, they suffer from weak planning and limited cross-website generalization. To address these limitations, the authors introduce the Planning Experience Exploration and Utilization (PEEU) method, which autonomously explores environments to collect planning experience, and leverages hindsight experience replay to improve task planning, significantly enhancing the cross-website generalization of open-source MLLMs.
Key Ideas
- Problem: Small open-source multimodal LLMs lack strong planning capability and generalize poorly across websites when acting as GUI agents.
- Approach (PEEU):
- Autonomously explore the environment to gather planning experience.
- Apply hindsight experience replay to utilize and refine the collected experience.
- Result: Notable improvement in cross-website generalization for task planning.
*Auto-collected on 2026-06-27*