ShopGym: A Digital-Twin Simulation and Benchmarking Framework for E-Commerce Web Agents

Forum topic · QianXun · 2026-05-19

Summary

ShopGym is a framework presented by researchers from Shopify and North Carolina State University (arXiv: 2605.16116) for realistic simulation and scalable benchmarking of e-commerce web agents, or computer-use agents (CUA). The framework combines two components: ShopArena, a "cloning protocol" in which a reconnaissance agent explores real seed storefronts and generates anonymized, self-contained sandbox replicas with local data that can be reset at will; and ShopGuru, a task-synthesis system that automatically generates thousands of shopping tasks ranging from simple actions (adding items to a cart) to complex multi-step journeys (comparing products by warranty and price). Evaluations show cloned sandboxes closely match original stores in action density and hierarchy depth, and that agent performance in ShopGym correlates with success on real web pages. The article also raises concerns: simplified handling of dynamic noise like pop-ups and CAPTCHAs, potential reward hacking due to mocked payments, and high LLM compute costs for generating sandboxes. ShopGym argues that future AI competitiveness depends as much on training infrastructure as on the models themselves.

ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents

Authors: Chinmay Savadikar, Mingyu Zhao, Tianfu Wu, et al. (Shopify, NC State University, etc.)
arXiv ID: 2605.16116 (May 2026)
Domain: Computer-use agents (CUA), web automation, simulation and evaluation
Keywords: ShopArena (simulator), ShopGuru (task synthesis), digital twins, behavioral correlation

The Problem

Training e-commerce web agents today faces a dilemma:

Real websites are chaotic: prices change, inventory runs out, pop-ups jump around — making standardized evaluation nearly impossible.
Existing simulators are too simplistic: agents can ace toy test sites but fail on real pages.

ShopGym aims to be the ideal middle ground: a "cyber training ground" that faithfully mirrors real shopping sites while remaining controllable, resettable, and safe.

Core Architecture

1. ShopArena — the "cloning factory"

sandbox replica

2. ShopGuru — the "exam generator"

Short tasks (simple skills): e.g., "add this red dress to the cart."
Long tasks (complex journeys): e.g., "compare three coffee makers, find the one with the longest warranty at the lowest price, and complete all steps up to checkout."

Evaluation Results

Structural fidelity: graph-based analysis shows cloned sandboxes closely match original stores in action density and hierarchy depth.
Behavioral correlation: agents that perform well in ShopGym simulations also achieve higher success rates on real web pages — meaning the simulated benchmark genuinely reflects real-world capability.

Open Questions and Criticisms

1. Missing "pop-up monsters": real e-commerce sites feature flash-sale pop-ups, CAPTCHAs, and personalized recommendations. ShopArena's cloning focuses on structural skeletons — dynamic, adversarial noise may be oversimplified. 2. Broken causality of money: all payments in the simulator are mocked. Could agents develop reward-hacking behaviors that fall apart when facing real payment interfaces without real monetary stakes? 3. Compute and scale: generating a full sandbox store consumes substantial LLM resources — potentially a high barrier for smaller organizations.

Takeaway

ShopGym demonstrates that AI's future competitiveness lies not just in models but in the training infrastructure behind them. Through digital-twin simulation and scalable task synthesis, it provides a faithful yet controllable proving ground for e-commerce web agents — grounding evaluation in "digitalized realism" while leaving open questions around dynamic fidelity, payment realism, and cost.