InstructMesh: Interactive Post-Generation Refinement of AI-Generated 3D Models for Fabrication

Forum topic · 小凯 · 2026-09-01

Summary

InstructMesh is an interactive post-generation refinement tool for repairing AI-generated 3D models intended for fabrication. While generative AI can create 3D models from text or images, these models prioritize visual plausibility over geometric accuracy, often producing defects that compromise real-world use. InstructMesh enables selective repair through region selection and targeted operations such as opening or sealing voids and adjusting local thickness. Users invoke edits via natural language prompts or slider controls, and by operating directly on an intermediate latent representation, robust geometric corrections are possible without expert modeling skills. The design was informed by an analysis of common fabrication-related failure modes in state-of-the-art generative tools. Two user studies demonstrated that novices can perform fabrication-related repairs on generative outputs, and revealed a preference for hybrid interfaces combining slider controls with natural language input. Authors include Faraz Faruqi, Stefanie Mueller, and collaborators from MIT and Google. arXiv:2608.28534.

Overview

Research area: Machine Learning Authors: Faraz Faruqi, Ahmed Katary, Demircan Tas, Theresa Hradilak, Ning Zhang, Jiaji Li, Fabian Manhardt, Martin Nisser, Vrushank Phadnis, Ruofei Du, Federico Tombari, Megan Hofmann, Stefanie Mueller Published: 2026-08-28 arXiv: 2608.28534

Problem

Recent advances in generative AI allow users to create 3D models from text or images. However, these models prioritize visual plausibility over geometric accuracy, often generating results with flaws that compromise their intended use post-fabrication.

Approach

InstructMesh is an interactive post-generation refinement tool that enables selective repair of generative 3D models through:

Region selection combined with targeted operations, such as opening or sealing voids, or adjusting local thickness
Natural language prompts or slider controls to invoke edit operations
Direct operation on the intermediate latent representation, enabling robust geometric corrections without expert modeling skills

Methodology and Findings

To inform the design, the authors first analyzed common fabrication-related failure modes in outputs of state-of-the-art generative tools. They then conducted two user studies showing that:

Novices can use InstructMesh to perform fabrication-related repairs on generative outputs
Users prefer a hybrid interface combining slider controls with natural language input