Crafter: A Multi-Agent Harness for End-to-End Scientific Figure Generation and Editing

*English summary of a forum post analyzing the Crafter paper by a UIUC / Tsinghua / Peking University joint team. Project code: https://github.com/HaozheZhao/Crafter*

Key points

• Scientific figures are structured information panels where misdirected arrows or overlapping labels break the figure. Existing AI tools are fragmented: static-only output, single figure type, or text-only input without sketch support.
• Crafter's core thesis: the bottleneck is not a stronger backbone model but a harness—an orchestration layer for planning, verification, and correction—centered on a shared structured specification.
• Three mechanisms matter most:
• Parallel exploration: K candidate layouts are generated and scored before refinement, escaping fundamental compositional errors early (removal costs 8.56 points).
• Structured correction: critic diagnoses are converted into typed edit operations written into the specification, then prompts are re-assembled—not appended—to avoid contradictory accumulation (removal costs 8.90 points, the largest ablation).
• Targeted criticism: the critic outputs six-dimension scores, specific defects, and suggested fixes; iteration runs at most 3 rounds with rollback to the best prior version (removal costs 5.48 / 5.04 points respectively).
• The pipeline uses five agents around the shared spec: intent reasoner, plan generator, image generation backend, critic, and specification refiner, plus a convergence judge for accept / continue / rollback decisions.

CraftEditor: from rendering to editing

CraftEditor converts raster figures (PNG/JPG) into editable SVGs using the same harness pattern:

1. Extraction: a VLM analyst plans keep/remove operations; an instruction-based editor produces a clean canvas, verified for up to 3 rounds. 2. Processing: elements are annotated, localized, and classified as vector or raster. 3. Composition: two candidate SVG skeletons are generated, the better one selected, elements inserted, then refined in a critic loop (max 4 rounds) combining VLM checks (layout fidelity, semantics) with programmatic audits (text overflow, arrow endpoint precision, overlap).

CraftEditor leads baselines across all 7 dimensions: 8.04 overall vs. 6.91 (Edit-Banana) and 3.69 (AutoFigure-Edit).

CraftBench: a less biased benchmark

Existing benchmarks each cover only one figure type or input mode (PaperBanana-Bench, SridBench, IGenBench, SciFlow-Bench). CraftBench spans 3 figure types × 4 input conditions:

• Text-to-figure (179 samples), mask completion (30), key-element composition (30), sketch-conditioned generation (40)
• Styles: academic figures (140), posters (109), infographics (30)
• Sources: arXiv papers from 18 research fields, award-winning conference posters, research blogs; 7-stage quality filtering plus 3-person unanimity human review

Results

• Best score on every dimension and task of both PaperBanana-Bench (+16.61 over the strongest agentic baseline) and CraftBench (+22.20).
• Swapping the backend from Nano Banana 2 to Nano Banana Pro only gains 0.34–2.10 points, indicating the harness's contribution is largely independent of generator capability—newer backends plug in without architectural changes.
• Ablating any mechanism drops 5.04–8.90 points; structured correction matters most, suggesting contradictory instruction accumulation is the top killer of scientific figure generation.

Why it matters

• A generalizable paradigm: structured orchestration over raw model strength applies to code, tables, charts, and slide generation.
• End-to-end loop: figures that can be generated, iterated, and reused—not just rendered once.
• Executor-agnostic: harness layers can swap Nano Banana, GPT-Image, or future models freely.
• Fairer evaluation: CraftBench fills the cross-type, cross-condition evaluation gap.

Limitations

• Critic and refiner are prompt-based LLMs, not trained vision models; fine visual details (small text, subtle arrow endpoints) can be missed.
• Raster-to-vector conversion still struggles with complex textures (gradients, shadows).
• Compute cost is high: K parallel plans × up to T rounds; trained lightweight critic models may help.
• Highly complex non-grid layouts (e.g., topology diagrams) still exhibit structural errors.

Bottom line

Crafter's core insight: making scientific figures is less about drawing *better* and more about drawing *methodically* and editing *surgically*—structured specifications instead of free-text prompts, targeted diagnoses instead of vague scores, and parallel exploration instead of blind retries.

Paper and code: https://github.com/HaozheZhao/Crafter