AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards

Forum topic · 小凯 · 2026-10-15

Summary

AesCode is a framework for generating information-rich visual artifacts such as slides, posters, and dashboards as code, addressing the problem that code models cannot see how their layout and color choices render on the canvas. AesCode grounds code generation in the content prompt while using an image generated from the same prompt as aesthetic guidance. It represents the target as a design graph describing the whole canvas, making each property attributable, and derives complementary cross-modal rewards from it: deterministic verifiers score parseable properties, while a VLM judge evaluates non-parseable properties using Visual Graph Rubrics (VGR) tied to graph elements. The decoupled rewards are optimized via reinforcement learning with per-channel normalization. On benchmarks scoring verifiable correctness and visual quality, AesCode-8B and AesCode-32B improve visual quality by 31.1 and 22.4 percentage points over a same-scale Qwen3-VL baseline with identical references, matching closed-source frontier models such as GPT-5.5 and Claude Opus 4.8. Paper: arXiv 2610.18860.

Paper Overview

Field: ML
Authors: Xin Gao, Yifan Yang, Xin Li, Pan Wang, Yan Li, Muzhao Tian, Mingxi Cheng, Bei Liu, Kai Qiu, Qi Dai, Dong Chen, Yue Dong, Leandros Tassiulas, Rex Ying, Ji Li, Chong Luo
Published: 2026-10-13
arXiv: 2610.18860

Abstract

Generating information-rich visual artifacts such as slides, posters, and dashboards as code is appealing because the outputs are structured, editable, and verifiable. Yet code models cannot see how their choices of layout and color come together on the canvas, often limiting spatial planning and aesthetic quality. Image generators excel at visual design but can misrender requested content, making their outputs unreliable targets for direct reproduction. We therefore introduce AesCode, which generates code grounded in the content prompt while using an image generated from the same prompt as an aesthetic guidance.

To do so, AesCode represents the target as a design graph that describes the whole canvas and makes each property attributable. From this graph, we derive complementary cross-modal rewards: deterministic verifiers score properties parseable from the code and rendered output, while a VLM judge evaluates non-parseable properties using sample-specific Visual Graph Rubrics (VGR) bound to graph elements and relations. We optimize these decoupled rewards with reinforcement learning using per-channel normalization, jointly improving spatial planning, aesthetic quality, and verifiable correctness.

Key Findings

To evaluate this setting, the authors build benchmarks that separately score verifiable correctness and visual quality; the visual score aligns well with human judgment.
Reference images significantly improve the visual quality of code generated by existing models.
Ablations show AesCode benefits from the reference image both as a model input and as a source of visual reward.
Relative to a same-scale Qwen3-VL baseline given the same references, AesCode-8B and AesCode-32B improve visual quality by 31.1 and 22.4 percentage points respectively, reaching scores comparable to closed-source frontier models such as GPT-5.5 and Claude Opus 4.8.

---

*Auto-collected on 2026-10-15.*