AgentGarten: Code Worlds for Evolving Agents — Neural Rendering Meets Simulation for Agent Training

Forum topic · 小凯 · 2026-10-12

Summary

AgentGarten is a research framework (arXiv 2610.12374) that builds real-time interactive training environments for AI agents by coupling simulators and game engines with a shared neural renderer. The simulation backends maintain persistent world state and execute program-defined interaction rules, while the renderer produces realistic visual observations from structured conditions exported through a common interface. To build the renderer, the authors adapt a pretrained video model to geometric conditioning and distill it using a proposed technique called Adversarial Forcing, which makes history prefilling differentiable via exact replay and adds adversarial supervision on real data, achieving real-time inference. Agents in AgentGarten perceive the world visually, interact in real time, and distill each episode's experience into playbooks that later agents inherit and refine. Empirical results show dramatic learning-efficiency gains: agents trained for only 4 episodes surpass what traditional reinforcement learning requires millions of episodes to achieve. Because new worlds can be written as code and rendered through the same interface, environments scale in number and difficulty alongside the agents, a step toward continually evolving agents through interactive experience.

AgentGarten: Code Worlds for Evolving Agents

Research area: Computer Vision (CV)

Authors: Jiawei Chi, Shangchen Miao, Zhiyuan Shi, Kailu Wu, Hanyang Wang, Weiliang Chen, Qiyu Dai, Jinshan Ren, Jun Gao, Mingsheng Long, Yueqi Duan, Jiangran Lyu, Jialong Wu, Fangfu Liu

Published: 2026-10-08

arXiv: 2610.12374

Overview

Interactive virtual worlds allow agents to learn through exploration and interaction. What agents can learn is bounded by the environments they practice in, which must be both faithful (consistent state, rules, and dynamics) and realistic (observations following real-world visual distributions). Achieving both across diverse worlds has remained a bottleneck. AgentGarten addresses this by coupling simulators and game engines with a shared neural renderer to build real-time interactive environments.

Key components

Simulation backends maintain persistent world state and execute program-defined interaction rules.
Shared neural renderer generates visual observations from structured conditions exported through a common interface.
Adversarial Forcing: a proposed distillation technique that makes history prefilling differentiable through exact replay, so losses on subsequent predictions backpropagate into how the renderer encodes previous observations. Adversarial supervision on real data further improves visual quality.
The renderer adapts a pretrained video model to geometric conditioning and is optimized for real-time inference.

Learning paradigm

Agents in AgentGarten perceive the world through visual observations, interact in real time, and distill each episode's experience into playbooks that subsequent agents inherit and refine.

Results

Empirical studies show substantial gains in learning efficiency: agents exceed the performance that traditional reinforcement learning reaches only after millions of episodes, using just 4 episodes of learning.

Because new worlds can be written as code and rendered through the same interface, environments scale in both number and difficulty alongside the agents — a step toward agents that continually evolve through interactive experience.

Paper: https://arxiv.org/abs/2610.12374