Crafter: An End-to-End Generate-and-Edit Pipeline for Scientific Figures

Anyone who has made scientific figures knows the pain: producing a usable figure can take longer than writing the methods section itself. You tweak colors, align arrows, resize fonts—over and over. Then your advisor says "this label is wrong," and with a flat PNG, you either redraw from scratch or hack at it in Photoshop.

Existing AI tools are also lopsided here: some only generate static images with no editing; some handle only one figure type (e.g., method diagrams); some accept only text prompts and ignore hand-drawn sketches.

Crafter, from a joint UIUC, Tsinghua, and Peking University team, addresses all three gaps at once.

Why scientific figures are harder than landscape images

A scientific figure is a structured information panel—label boxes, arrows, icons, annotations—where every element has precise semantics and positional relationships. A mispointed arrow breaks the argument; overlapping labels break readability.

This creates three real technical difficulties:

1. High generation variance. The same prompt yields different layouts each time; errors appear in different places on every roll. 2. Free-text edits become self-contradictory. Appending instructions like "enlarge the title" and "reduce whitespace" to the prompt lets the model silently absorb conflicting requirements and produce a mess. The problem is the modification mechanism, not model strength. 3. Non-actionable feedback. A "5/10 quality" score tells the model nothing. It needs a concrete diagnostic: "arrow A's endpoint is 3 px off its target box," "label B overlaps icon C."

Crafter's approach: a harness, not a stronger engine

The authors' key judgment: scientific figure generation needs not a stronger backbone but a harness—an orchestration layer around the generation engine handling planning, verification, and correction.

The core is a shared structured specification that five agents operate on:

• Intent reasoner: analyzes inputs—text, sketches, reference images—to extract what the figure must convey
• Plan generator: proposes multiple candidate layouts (banner, multi-column grid, layered) at once
• Image generation backend: renders the chosen plan
• Critic: outputs concrete diagnoses—what's wrong, why, how to fix—not just scores
• Specification refiner: converts diagnoses into typed edit operations (e.g., "add constraint: center title", "forbid: decorative icons", "resize: element A to 120%") written back to the shared specification

A convergence judge decides when to accept, keep iterating, or roll back to the best previous version.

The crucial design: all edits are typed operations on the specification, not free-text appended to prompts, which fundamentally prevents contradiction accumulation.

Three mechanisms

1. Parallel exploration to pick the right framework first

The plan generator proposes K candidates with different layouts/styles, rendered in parallel; the critic picks the best for refinement. If the first figure has a fundamental compositional flaw (e.g., a comparison grid drawn as a flowchart), all later edits just patch a broken frame. Ablation: removing parallel exploration drops scores by 8.56, with readability hit hardest.

2. Structured correction instead of prompt stacking

Traditional loop: prompt → generate → append "enlarge labels" → regenerate → append "move arrow left" → … the prompt grows and contradictions pile up.

Crafter instead writes structured operations into an independent specification data structure; each round's prompt is reassembled from that structure. It's like maintaining a database of atomic operations rather than stacking sticky notes on an article.

Ablation: switching to free-text correction drops 8.90—the largest drop of all—showing contradiction accumulation is the number-one killer in scientific figure generation.

3. Targeted criticism that prescribes, not just scores

The critic outputs: six-dimension sub-scores (content faithfulness, readability, layout, formatting, etc.), identified defects, suggested corrections, and a corrected figure description. The loop runs at most 3 rounds, rolling back if a round regresses—LLM-driven iteration is not monotonic.

Ablation: removing the loop costs 5.48; removing targeted criticism costs 5.04.

CraftEditor: from "generated" to "editable"

Crafter generates raster images, but researchers often need to edit labels, colors, or layouts. CraftEditor converts raster figures into editable SVG using the same harness pattern:

• Extraction: a VLM analyst makes a keep/remove plan; an instruction-based image editor produces a clean canvas; the critic verifies over up to 3 rounds
• Processing: each element is annotated, localized, and classified as vector or raster
• Composition: two candidate SVG skeletons are generated, the better one chosen, elements inserted, then a critic loop runs—at most 4 rounds with rollback. The critic is hybrid: a VLM checks global layout fidelity and semantic correspondence, while programmatic checkers audit text overflow, arrow endpoint precision, and element overlap

CraftEditor outperforms existing baselines (Edit-Banana, AutoFigure-Edit) across all 7 evaluation dimensions: 8.04 overall vs. 6.91 and 3.69.

CraftBench: a benchmark that isn't lopsided

Existing benchmarks each have blind spots: PaperBanana-Bench (text→method figure only), SridBench (text input only), IGenBench (one figure type), SciFlow-Bench (flowcharts only).

CraftBench covers 3 figure types × 4 input conditions:

• text→figure (179 samples)
• mask completion (30)
• key element composition (30)
• sketch-conditioned generation (40)

Styles: academic figures (140), posters (109), infographics (30). Data comes from arXiv papers across 18 research fields, award-winning conference posters, and research blogs, filtered through a 7-stage pipeline plus 3-person unanimity human review.

Experimental results

• Crafter achieves the highest scores on both PaperBanana-Bench (+16.61 over the strongest agentic baseline) and CraftBench (+22.20), leading on every dimension and task
• Swapping the backend from Nano Banana 2 to Nano Banana Pro improves scores by only 0.34–2.10, showing the harness's contribution is largely independent of generator capability—stronger future models can plug in without architecture changes
• Ablations: removing any mechanism costs 5.04–8.90 points; structured correction matters most (-8.90)

Why this paper matters

1. A more general paradigm: structured generation problems (code, tables, charts, slides) may need orchestration harnesses rather than just better diffusion models 2. End-to-end closed loop: not just a pretty figure, but one that can be edited, iterated, and reused 3. Executor-agnostic: swap Nano Banana for GPT-Image or anything else without changing the architecture 4. Balanced evaluation: CraftBench fills the cross-type, cross-condition evaluation gap

Limitations

• Critic and refiner are prompt-based LLMs, not trained vision models; fine visual details (small text, subtle arrow directions) may be missed
• Vector conversion still struggles with complex textures (gradients, shadows)
• Compute cost is high: K parallel plans plus up to T loop rounds; a trained lightweight critic model could help
• Failure cases: complex non-grid-aligned topological layouts still show structural errors

One-line takeaway

Crafter's core insight: making scientific figures isn't about making the model "draw better" but "draw with method and edit with purpose"—structured specifications replace free-text prompts, targeted criticism replaces vague scoring, and parallel exploration replaces blind retries.

Paper and code: https://github.com/HaozheZhao/Crafter