Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design (arXiv 2609.22086)

Forum topic · 小凯 · 2026-09-22

Summary

Designer-RSI (arXiv 2609.22086) is a continual adaptation framework for professional graphic design, a long-horizon agentic task lacking reliable programmatic oracles. A frozen frontier model operates professional design software via more than 230 tools, while an external procedural memory of natural-language skills accumulates and refines reusable design procedures. The memory widens by acquiring procedures for uncovered recurring subtasks and deepens by revising existing procedures against their own successes and failures; a matched replay gate admits only changes that fix failures without regressing prior successes. Over five iterations processing 1,406 real user briefs and 1,869 auto-graded trajectories—without weight updates or human labeling—the skill library grew from 76 to 139 skills, raising Claude-Sonnet-4's GenEval2 execution success rate from 72.7% to 99.3% (+11.99 generation quality). Against skill-less agents it achieved win rates of 61.8% (Claude-Sonnet-4) and 67.6% (Claude-Opus-4.6) on four professional design benchmarks. Combining both mechanisms reached 58.5% win rate versus ~49% for either alone (p = 0.025).

Overview

Field: CV Authors: Hongyang Du, Lan Yan, Christian Flores, Asim Kadav Published: 2026-09-18 arXiv: 2609.22086

Abstract (full translation)

Professional graphic design is a long-horizon agentic task in which structured, editable artifacts emerge from many interdependent actions, yet outcomes admit no reliable programmatic oracle. We introduce a continual adaptation framework in which a frozen frontier model operates professional design software through more than 230 tools, while an external procedural memory of natural-language skills accumulates and refines reusable design procedures from experience.

The memory widens by acquiring procedures for recurring uncovered subtasks and deepens by revising existing procedures against their own successful and failed executions. A matched replay gate admits only changes that repair failures without regressing observed successes.

Results

Five iterations over 1,406 real user briefs and 1,869 auto-graded trajectories, with no weight updates and no human labeling
Skill library grew from 76 document-derived skills to 139
Claude-Sonnet-4 execution success rate on GenEval2 improved from 72.7% to 99.3% (generation quality +11.99)
Win rate over a skill-less agent: 61.8% (Claude-Sonnet-4) and 67.6% (Claude-Opus-4.6) across four professional design benchmarks
Ablation on 200 held-out briefs: widening-only 49.4%, deepening-only 48.6%, combined 58.5% (p = 0.025)

Conclusion

Procedural memory offers a practical path for continual agent adaptation under noisy, unverifiable feedback.

--- *Auto-collected on 2026-09-22.*