Subgoal-aware Memory Graph: Finding the Right Grain of Experience for GUI Agents

This post is a long-form Chinese review of *Subgoal-aware Memory Graph for GUI Agent* (Huawei Noah's Ark Lab, arXiv:2610.31307). Below is a structured English summary of its argument and findings.

Key points

• The granularity problem. GUI agents face a dilemma in what they remember: action-level memories (each tap/swipe/type) are too fine to carry task semantics, while task-level trajectories are too coarse—bound to one task and non-reusable across tasks that share intermediate steps but diverge in details.
• Subgoal as the right unit. The paper proposes the subgoal: a contiguous sequence of actions sharing one intent (e.g., "open the search bar and submit a query", "add the selected sunglasses to the cart"). Using a kitchen analogy, it is the equivalent of "blanch the pork belly to remove gaminess"—coarse enough to hold intent, fine enough to transfer across dishes (or apps).
• Experience cards. Each subgoal is annotated as g = ⟨s_from, s_to; text, effect, behaviour, success⟩: two boundary screenshots, an intent summary, a one-line description of the visual change, a concise behavior summary, and a success flag ∈ {yes, no, uncertain} labeled by an LLM judge. Middle frames are discarded—start and end suffice.
• Memory graph construction (three steps):
• State merging: nodes are screen states; a new screenshot joins an existing node if embedding cosine similarity ≥ τ = 0.9, otherwise a new node is created. Experiences from different tasks flowing through the same "search page" converge on the same node.
• Edges: each subgoal becomes a directed edge from source to destination state, carrying the full five-part annotation.
• Deduplication: under the same source node, edges with the same target, same success label, and intent-text embedding similarity > θ = 0.9 are treated as duplicates; the older edge is kept.
• Retrieval: encode the current screenshot, find the nearest node (top-k with k = 1), and inject all outgoing edges—successes and failures alike—into the prompt as a memory block. Extra latency is ~0.5 seconds. Returning failed subgoals lets the agent know where traps lie, not just where the road goes.

Experimental results (SPA-Bench, English single-app level-3, 13 tasks, 3 runs averaged; backbone Qwen3-VL-8B-Instruct, embeddings Qwen3-VL-Embedding-8B)

| Memory grain | Structure | Success rate | |---|---|---| | None | — | 34.19% | | Subgoal | Flat (no graph) | 30.77% | | Action | Graph | 38.46% | | Task | Graph | 36.75% | | Subgoal (proposed) | Graph | 45.30% |

• +11.11 points over the no-memory baseline.
• The striking 30.77%: well-cut subgoals *without* graph organization perform worse than no memory at all—unaggregated similar cards become retrieval noise. Good granularity cannot rescue bad organization.
• With a graph, action-level (38.46%) and task-level (36.75%) memory still trail subgoal-level by over six points: the graph amplifies existing semantics but cannot create them.
• Engineering cost: the full graph takes 4 GPU-hours to build; each retrieval adds ~0.5 s.

Interpretation: granularity × structure is a multiplication, not an addition

The reviewer highlights that this is effectively a factorial (ablation) design: grain (action/subgoal/task) × structure (flat/graph). The value of granularity is *conditional*—it must live inside an organizing structure to pay off; conversely, the graph's value depends on how much semantics the edges already carry. A deeper reframing: the graph's nodes record situations ("what the world looks like here"), not behaviors or completed tasks. Situations outlive tasks, so experience attached to them survives across tasks.

Honest limitations

• The method assumes the GUI environment approximates a discrete state space; highly dynamic interfaces (e.g., video players) break the screenshot-similarity proxy.
• The 13-task benchmark is small; absolute success rates are backbone-sensitive.
• Subgoal segmentation and success judgments rely on LLMs, whose errors propagate into the graph; robustness here is not analyzed.

Context

This is framed as the second in a trilogy on agent memory management: the first (Muffin) addressed inter-agent memory sharing at the "discovery" grain; the third (teased) will ask how much memory is worth when acquiring, storing, and retrieving it all have costs.

References

• Luo, D., Beechey, D., Yuen, D., & Luo, W. (2026). *Subgoal-aware Memory Graph for GUI Agent*. arXiv:2610.31307 [cs.AI]. Huawei Noah's Ark Lab.
• Benchmark: SPA-Bench English single-app level-3 split (13 tasks, 3 independent runs)
• Backbone: Qwen3-VL-8B-Instruct; embedding model: Qwen3-VL-Embedding-8B
• Hyperparameters: visual merge threshold τ = 0.9, text dedup threshold θ = 0.9, retrieval k = 1
• Build cost: 4 GPU-hours; retrieval latency: ~0.5 s