Do GUI Agents Know When Not to Act? Enabling Conflict-Aware Termination for Multimodal GUI Agents
研究领域: AI/ML 作者: Zhaoyuan Huang, Tianjie Ju, Pengzhou Cheng 发布时间: 2026-09-06 arXiv: 2509.00009
论文概要
研究领域: AI/ML 作者: Zhaoyuan Huang, Tianjie Ju, Pengzhou Cheng 发布时间: 2026-09-06 arXiv: 2509.00009
中文摘要
图形用户界面(GUI)智能体越来越多地用于在用户界面上执行自然语言指令，但真实用户可能因善意错误而发出不可行的指令。一个可靠的智能体不仅应该知道如何行动，还应该知道何时不行动。在本工作中，我们引入了CONFLICTGUI，一个涵盖指令内部冲突和指令-GUI上下文冲突的基准测试，用于研究冲突感知终止。我们的评估揭示了严重的执行偏向过度服从：在可行任务上表现良好的智能体在冲突指令下往往继续盲目执行。为了缓解这种行为，我们提出了CONFLICTGUARD，一种推理时框架，将智能体的可行性感知与其动作生成对齐。CONFLICTGUARD包含两个耦合组件：可行性验证协议，指导智能体在行动前评估指令逻辑和GUI端证据；以及条件动作调节机制，将智能体从过度服从执行引导到终止导向行为。在五个广泛使用的智能体上的实验表明，CONFLICTGUARD显著提高了平均冲突任务成功率，同时保持了正常的GUI任务性能。这些结果验证了轻量级推理时干预可以大幅提升GUI智能体识别不适当执行场景并避免不必要动作的能力。
原文摘要
Graphical user interface (GUI) agents are increasingly used to execute natural-language instructions on user interfaces, yet real users may issue infeasible instructions due to benign mistakes. A reliable agent should not only know how to act, but also when not to act. In this work, we introduce CONFLICTGUI, a benchmark covering instruction-internal conflicts and instruction-GUI context conflicts to study conflict-aware termination. Our evaluation reveals severe execution-biased overcompliance: agents that perform well on feasible tasks often continue to execute blindly under conflicting instructions. To mitigate this behavior, we propose CONFLICTGUARD, an inference-time framework that aligns an agent's feasibility awareness with its action generation. CONFLICTGUARD contains two coupled co...
*自动采集于 2026-09-06*
#论文 #arXiv #AI #小凯