AssemState Benchmark: Can AI Assemble IKEA Furniture from Manual Instructions?

Forum topic · 小凯 · 2026-10-07

Summary

AssemState is a benchmark from Tsinghua University, Xidian University, and UIUC that tests whether multimodal LLMs can assemble furniture from IKEA-style manuals. It splits the task into two independent exams: recovering the assembly tree (structural planning) and predicting the 6D pose of each part (geometric execution). With GPT-5 as the backbone, the proposed anchor-guided pipeline reaches 62.80 F1 and 53.92% exact tree match on structure recovery, clearly beating prior methods like Manual2Skill++ (38.58 F1). Pose reasoning remains far weaker: a render-diagnose-refine loop improves chamfer distance (5.41 to 1.77) but judge acceptance is only 5.3%, PA@0.01 is zero, and translation/rotation errors barely move. Simulation tests in IsaacLab show ~60% of assemblies collapse on release. The paper's failure taxonomy covers collision, floating, wrong-side, mirror, and not-fully-seated errors, and it is notable for explicitly stating its evaluation boundaries, showing that structural planning is largely solved while target-free 6D pose estimation is not.

Overview

This post is the third installment of a "Body and Senses" series evaluating AI on physical-world tasks. Previous entries covered physical intuition (WorldSolver) and conversational timing in full-duplex speech models (Coupled but Late). Today's subject: hands — can an MLLM read an IKEA manual and actually assemble furniture?

The answer comes from AssemState: Manual and Physical-State-Guided Reasoning for Zero-shot Furniture Assembly (arXiv: 2610.08446), by Zhiyuan Qi, Jierui Li (co-first authors), Yifan Shen, Cheng Qian, and Jiateng Liu, from Tsinghua University, Xidian University, and UIUC, submitted to the Physical World AI Workshop.

What the benchmark tests

Two distinct capabilities are measured on 102 real IKEA products:

1. Assembly tree recovery — reconstructing the hierarchical merge structure (leaves = parts, internal nodes = sub-assemblies). Crucially, manual pages ≠ tree nodes ≠ atomic operations: 85/102 products have multi-way merge nodes, 18 have merges between two non-leaf sub-assemblies. 2. 6D pose reasoning — converting semantic relations ("shelf attaches to left wall") into position + orientation, e.g., drawer panels misaligned by a centimeter or mirrored symmetric panels that look identical on paper.

Honest experimental design

The two exams are deliberately decoupled:

Structure recovery: given numbered scenes, manual pages, and *pre-computed* step masks (mask generation is explicitly out of scope).
Pose reasoning: conditioned on the *correct* current atomic operation and prior correct state — no end-to-end error propagation, no target pose exposed.

This isolates planning errors from geometric errors, at the cost of measuring "decomposed moves" rather than "the whole routine."

Method

Structure recovery (three stages): (1) resolve each numbered part into name/function, filtering out tools like screwdrivers; (2) anchor-guided planning — one MLLM call finds the first and last steps as boundary anchors, assigns parts to steps, and emits a textual plan; (3) rule-based parsing into a nested-integer-list tree.json.

Pose reasoning: propose a candidate pose → render from multiple viewpoints → a diagnostic judge returns accept/reject, error type, and correction cues; rejected poses get an incremental correction δT (left-multiplied), for three rounds. Error types map to fix directions: wrong_side/wrong_mirror → large reorientation; floating → translate along contact normal; collision → back off along anti-normal; not_fully-seated → small insertion translation.

Results

Structure recovery (GPT-5 backbone):

| Method | Precision | Recall | F1 | Exact tree match | |---|---|---|---|---| | SingleStep | 10.78 | 10.78 | 10.78 | 10.78 | | GeoCluster | 13.37 | 13.36 | 13.15 | 3.92 | | Manual2Skill | 37.45 | 36.41 | 36.83 | 27.45 | | Manual2Skill++ | – | – | 38.58 | 28.24 | | No anchors | 57.39 | 55.90 | 56.20 | 45.10 | | Ours | 62.34 | 62.46 | 62.80 | 53.92 |

Ablation shows anchoring alone contributes ~6 F1 points and ~9 points of exact match.

Pose reasoning (243 atomic operations, 55 correct trees):

| Setting | Judge acceptance ↑ | Trans. RMSE ↓ | Rotation (°) ↓ | Chamfer (mean/median) ↓ | |---|---|---|---|---| | Text-only | 0.0 | 2.4322 | 123.02 | 15.7263 / 10.1914 | | Open6DOR-style | 0.0 | 1.8747 | 120.84 | 5.4111 / 0.9391 | | Ours (3-round self-check) | 5.3 | 1.8898 | 121.52 | 1.7744 / 0.8112 |

Key observations:

Chamfer distance improves substantially (5.41 → 1.77); weaker models benefit more (Claude Haiku 4.5: acceptance 4.5 → 11.5, median chamfer 3.2725 → 1.8770).
But PA@0.01 is zero — no attempt passes the strictest tolerance.
Translation RMSE and rotation error barely move across refinement rounds: the loop fixes local surface contact, not absolute target-free 6D pose estimation. The paper says this explicitly.

Simulation (IsaacLab): stability = bbox gap < 0.05, vertical drop < 0.05, tilt < 12°. GPT-5 release stability rose from 37.0% to 39.5%; Claude pinned at 33.7%. Ablation reveals a trade-off: multi-view initialization buys stability (44.9%), contact-aware self-check buys contact quality but stability drops back to 39.5%. The paper explicitly refuses to claim all metrics improve monotonically.

Five failure modes

All rejected assemblies fall into: collision (interpenetration), floating (no support), wrong-side (correct region, wrong face), mirror (flipped orientation), and not-fully-seated (missing final insertion). GPT-5.2 errs mostly on wrong-side; Haiku 4.5 on floating/mirror. One telling trace: op-0 of Bench/applaro was rejected three times, always wrong_side — the loop faithfully redirects the pose each round, but visual reasoning keeps re-selecting the same wrong face.

Stated limitations

Mask generation not evaluated; pose evaluation conditioned on correct operations/states (no end-to-end testing).
Judge acceptance is model-assisted diagnosis with no human verification; proposer, selector, and judge are all GPT-5.2 (same-family evaluator bias).
Simulation is a coarse stability probe — no fastening, insertion constraints, mesh-level contact, gripper interaction, or collision-free motion testing.

Takeaway

The closing synthesis: across the trilogy (skin = physics, ears = timing, hands = assembly), AI fails all three exams — but with decimal-point-precise diagnoses. Structure planning is ~60% solved; target-free 6D geometric execution is not solved at all. That gap marks exactly where the next round of work should begin. Intelligence lives in a body, not just in symbols.

Reference: AssemState: Manual and Physical-State-Guided Reasoning for Zero-shot Furniture Assembly, arXiv 2610.08446.