The Semantic-Execution Gap in GUI Agents
Even when an AI agent *understands* the correct procedure, it can still fail catastrophically on precision tasks. On ordinary apps, clicking a 48-pixel-wide button tolerates slight aim errors. In CAD-style geometric software, however, a one-pixel mistake propagates: if the first step's anchor point is offset by one pixel, every dependent construction (tangent lines, intersections, etc.) breaks, and the task collapses.
Benchmarking on GeoGebra, the authors found a striking paradox:
- Intent accuracy: ~88% — the agent knows which button and which endpoints to select.
- End-to-end success: <6% — topological dependency errors cascade into total failure.
- Distance penalty — a click 2 pixels off sharply reduces the reward; 5 pixels off is treated as failure.
- Geometric consistency check — after drawing, the system verifies with mathematical predicates (e.g., *is this segment truly tangent to the circle?*) and rewards only when the relation actually holds.
This mismatch is what the paper calls the Semantic-Execution Gap.
PAGER: Two Mechanisms for Pixel-Precise Control
1. Dependency-Structured Planning
Before acting, PAGER builds an explicit *component dependency graph*. Line A must depend on points B and C, and the agent is forced to execute in topological order. New operations are anchored to already-verified primitives rather than to floating coordinates.
2. Precision-Aligned Reinforcement Learning
Standard RL rewards any click "close enough." PAGER replaces this with strict geometric feedback:
Results
On the PAGE benchmark, PAGER's complex-task success rate rose from under 9% to 62%, more than a 4x improvement. This level of accuracy begins to make AI viable for CAD drafting, circuit design, and other domains where "off by a hair, off by a mile" is the rule.
Open Questions
1. Inference cost — Maintaining a dependency graph plus per-step pixel verification adds compute. The paper offers limited discussion of latency and whether real-time industrial pipelines can absorb the overhead. 2. 3D extension — PAGER is validated primarily on 2D GeoGebra. Dependency relations explode combinatorially in full 3D modeling, raising the question of whether the approach scales or suffers a *dimensional collapse*. 3. Mid-task disruption — If a human user manually moves an anchor point and breaks the dependency chain, does PAGER self-heal, or fall into a logical loop?
Key Takeaway
Real intelligence is not just grand intent; it is mastery of the smallest details. PAGER injects topological intuition into mechanical action, demonstrating that an agent can move from "keyboard theorist" to "digital craftsman." Truth lives in the details, and success is measured in pixels.