AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents

Forum topic · 小凯 · 2026-10-02

Summary

AssemblyWorld is an interactive 3D environment and benchmark designed to test whether pretrained general-purpose agents can assemble objects through visual interaction alone, without assembly-specific fine-tuning. Agents inspect rendered 2D views and manipulate supplied rigid parts—optionally guided by reference images or assembly manuals—rather than accessing mesh vertices or faces, and their assemblies are evaluated geometrically. The accompanying AssemblyWorldBench contains 100 assembly tasks across 80 objects spanning furniture, industrial assembly, and fracture reassembly. Evaluation of eight agent systems reveals notable capability gaps: the strongest system achieved 80.9% part accuracy but only 59.4% full-assembly success rate, while open-source systems lagged far behind stronger closed-source counterparts in both reliability and assembly accuracy. Analysis of visual references, interaction trajectories, and failure cases shows agents retain localization errors even while correcting assemblies. Paper: arXiv 2609.26761.

Paper Overview

Field: Computer Vision (CV)
Authors: Jiahao Zhang, Yeying Fan, Moitreya Chatterjee, et al.
arXiv: 2609.26761

Abstract (translated from the Chinese summary)

The task of 3D assembly requires translating an understanding of parts and their relationships into precise spatial arrangements. Can pretrained general-purpose agents assemble objects through visual interaction without additional assembly-specific fine-tuning? To investigate this question, the authors introduce AssemblyWorld, an interactive 3D environment in which agents inspect rendered views and manipulate supplied rigid parts, guided by images or assembly manuals when available. Agents perceive part geometry through 2D views rather than direct access to mesh vertices or faces, while their resulting assemblies are evaluated geometrically.

Building on this environment, the authors construct AssemblyWorldBench, comprising 100 assembly tasks across 80 objects spanning three domains: furniture, industrial assembly, and fracture reassembly.

Key Findings

Evaluation of eight agent systems reveals significant differences in capability.
The strongest system achieves 80.9% part accuracy, but only a 59.4% full-assembly success rate.
Evaluated open-source systems lag substantially behind stronger closed-source peers in both execution reliability and assembly accuracy.
Analysis of visual references, interaction trajectories, and failure cases shows that agents retain localization errors even while correcting assemblies.

Significance

AssemblyWorld provides a general-purpose platform for evaluating interactive assembly agents and for characterizing the gap between approximate structural recovery and precise reconstruction.

---

*Auto-collected on 2026-10-02. Source: zhichai.net forum post.*