Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generation

Forum topic · 小凯 · 2026-03-19

Summary

Omni-I2C is a comprehensive benchmark for evaluating Large Multimodal Models (LMMs) on image-to-code generation: converting complex, structured digital graphics into executable code. The paper argues this task requires an unprecedented synergy between high-fidelity visual perception and precise code generation, where any minor perceptual hallucination or coding error causes complete visual reconstruction failure. The benchmark contains 1080 meticulously curated samples spanning diverse subjects, image modalities, and programming languages, including real user-sourced cases from scientific visualizations to complex symbolic representations, each paired with executable reference code. Its evaluation framework decouples performance into perceptual fidelity and symbolic precision, moving beyond surface-level accuracy to reveal fine-grained structural failures and reasoning bottlenecks. Evaluations show significant performance gaps among leading LMMs; even state-of-the-art models struggle to maintain structural integrity in complex scenarios, indicating that multimodal code generation remains a formidable challenge. Data and code are available at https://github.com/MiliLab/Omni-I2C.

Overview

We present Omni-I2C, a comprehensive benchmark designed to evaluate the capability of Large Multimodal Models (LMMs) in converting complex, structured digital graphics into executable code. We argue that this task represents a non-trivial challenge for the current generation of LMMs: it demands an unprecedented synergy between high-fidelity visual perception -- to parse intricate spatial hierarchies and symbolic details -- and precise generative expression -- to synthesize syntactically sound and logically consistent code. Unlike traditional descriptive tasks, Omni-I2C requires a holistic understanding where any minor perceptual hallucination or coding error leads to a complete failure in visual reconstruction.

Key Details

Field: Computer Vision (CV)
Authors: Jiawei Zhou, Chi Zhang, Xiang Feng
Published: 2025-03-18
arXiv: 2503.13829

Omni-I2C features 1080 meticulously curated samples, defined by its breadth across subjects, image modalities, and programming languages. By incorporating real user-sourced cases, the benchmark covers a wide range of digital content -- from scientific visualizations to complex symbolic representations -- each paired with executable reference code.

Evaluation Framework

To complement this diversity, the evaluation framework provides the necessary depth: by decoupling performance into perceptual fidelity and symbolic precision, it goes beyond surface-level accuracy and reveals fine-grained structural failures and reasoning bottlenecks in current LMMs.

Findings

Significant performance gaps exist among leading LMMs on this task.
Even state-of-the-art models struggle to maintain structural integrity in complex scenarios.
Multimodal code generation remains a formidable challenge.

Resources

Data and code are available at: https://github.com/MiliLab/Omni-I2C