JD JoyAI-Image Deep Dive: Unified 8B+16B Architecture Awakens Spatial Intelligence, LongText-Bench 0.963 Dual SOTA in English and Chinese

Forum topic · 小凯 · 2026-06-14

Summary

JoyAI-Image from JD is a unified multimodal model combining an 8B Qwen3-VL-based MLLM for understanding and a 16B MMDiT diffusion engine for generation and editing, bridged by Wan-2.1-VAE. A key innovation is OpenSpatial, an automated data engine that lifts 3D oriented bounding boxes from 3D scans and in-the-wild web videos into 3M spatial QA pairs covering measurement, relations, camera awareness, multi-view consistency, and scene reasoning. With an 11.3M-example training mix, decoupled learning rates, and selective KL distillation, JoyAI-Image matches Gemini-2.5-Pro on spatial understanding (64.4 average) while remaining competitive on general benchmarks. On LongText-Bench it scores 0.963 in both English and Chinese text rendering, surpassing GPT Image 1 (0.619 Chinese) and Qwen-Image, and achieves 94.2 composition mean on T2I-CoReBench plus a 74.7% spatial editing accuracy gain on SpatialEdit-Bench. JD has open-sourced code, weights, the OpenSpatial-3M dataset, and Diffusers/ComfyUI support, positioning the model as a foundation for VLA and world-model research.

> Spatial intelligence is not icing on the cake — it is the foundation of unified vision models. JD's JoyAI-Image pairs an 8B understanding brain with a 16B generation engine, tying understanding, generation, and editing into one rope, showing that "seeing" and "drawing" can feed each other.

Key points

Unified architecture: JoyAI-Image argues understanding, generation, and editing are three facets of one visual intelligence, not three tasks to be stitched together. The MLLM's last-layer hidden states serve as the primary conditioning signal for the diffusion model, so generation is guided by understanding throughout.
Components: 8B MLLM (Qwen3-VL-8B-Instruct) + Wan-2.1-VAE (pixel-to-latent bridge, preserving fine text detail) + 16B self-developed MMDiT for denoising, synthesis, and editing.
Spatial intelligence via data: The OpenSpatial engine lifts 3D oriented bounding boxes from ScanNet/Matterport3D/ARKitScenes plus web videos, validated by multi-view cyclic consistency, producing OpenSpatial-3M (3M entries) across five capabilities: spatial measurement, spatial relations, camera perception, multi-view consistency, and scene-aware reasoning.
Training mix (11.3M total): 54.25% general understanding, 29.65% spatial understanding (OpenSpatial 3.3M + VST 49.4K), 11.98% instruction augmentation, 1.21% spatial editing, rest internal data — preserving generalist ability while boosting spatial skills.
Learning details: Decoupled learning rates (vision encoder 5e-6 vs 5e-5); KL distillation applied only on general data to avoid constraining new spatial knowledge; dynamic sequence packing with Flash Attention.

Benchmark results

Spatial understanding (13 benchmarks)

| Model | Spa. Avg | |---|---| | Gemini-2.5-Pro | 64.4 | | JoyAI-Image-Und | 64.4 (matches Gemini) | | Qwen3-VL-8B (baseline) | 59.1 (+5.3) | | GPT-4o | 57.7 |

Notable: VSI-Bench 60.1, AllAnglesBench 61.0, BLINK improved from 49.5 to 61.0 (+11.5), while general benchmarks (MMBench, MMStar, OCRB) remain competitive.

Text-to-image (LongText-Bench)

| Model | EN | ZH | |---|---|---| | JoyAI-Image | 0.963 | 0.963 | | GPT Image 1 (High) | 0.956 | 0.619 | | Qwen-Image | 0.943 | 0.946 | | Seedream 3.0 | 0.896 | 0.878 |

Also SOTA Word Accuracy 0.8739 on CVTG-2K, aided by OCR-aware captioning and dedicated text-rendering SFT data.

Composition vs reasoning (T2I-CoReBench)

JoyAI-Image: composition 94.2 (best), reasoning 55.9, overall 68.7
GPT Image 1 (High): composition 79.8, reasoning 69.0, overall 72.6
Qwen-Image: composition 83.7, reasoning 51.7, overall 62.4

Editing

Spatial editing accuracy on SpatialEdit-Bench improves +74.7% over baseline, powered by a three-part editing data engine (general, open-domain, spatial editing with static object transforms and dynamic camera viewpoint changes).

Engineering highlights

Five-stage data filtering: progressively tightened 208p → 512p → 1024p stages with aesthetic-score thresholds and cascaded IQA (statistics + NIQE/CLIP-IQA/MUSIQ), 90% human-verified accuracy.
Four-level captioning: short, long, extended-long, and structured JSON (subject, background, style, composition, visible text) with OCR-aware token fusion and filtering.
Label rebalancing over 285K tags: inverse-logarithmic downsampling of head categories, +20–50% boost for weak categories, taxonomy-aware diversity sampling.
Human scoring: aesthetics 50% / information density 30% / style purity 20%, with sentinel samples and batch rework rules.
Million-scale multi-view generation in Blender: ray-cast camera placement, dual prompts (dense caption + structured JSON).
Four-stage training: pre-training (208p→512p→1024p, flow matching) → continued training → SFT → RL (Flow-GRPO) with aesthetic and text-image-alignment reward models.

Downstream and open source

The paper demonstrates "Thinking with Novel Views" (reasoning from new perspectives) and "Reconstruction with Novel Views" — core capabilities for VLA models and world models.

Open-sourced: paper (arXiv:2605.04128), code at https://github.com/jd-opensource/JoyAI-Image, understanding and editing model weights on Hugging Face/ModelScope, OpenSpatial-3M and SpatialEdit datasets, Diffusers support (merged 2026.05.08), and ComfyUI support. Text-to-image and distilled versions are listed as upcoming.

Limitations

Heavy inference cost: 16B MMDiT + 8B MLLM.
Abstract reasoning still trails GPT Image 1 on T2I-CoReBench reasoning.
The flagship text-to-image and distilled versions are not yet released.

---

Paper: JoyAI-Image: Awaking Spatial Intelligence in Unified Multimodal Understanding and Generation. Lin Song et al., Joy Future Academy, JD. arXiv:2605.04128.

GitHub: https://github.com/jd-opensource/JoyAI-Image