JD JoyAI-Image Deep Dive: Unified Understanding, Generation, and Editing with Spatial Intelligence
> Spatial intelligence is not a bonus feature—it is the foundation of unified vision models. JD's JoyAI-Image pairs an 8B understanding brain with a 16B generation engine, fusing understanding, generation, and editing into one system, proving that "seeing" and "drawing" can feed each other.
1. Why Understanding, Generation, and Editing Must Be One Model
Existing multimodal models fall into roughly three camps:
- Understanding-focused: MLLMs handle VQA, OCR, and spatial reasoning, but cannot draw
- Generation-focused: diffusion models do text-to-image and image-to-image, but don't "understand" what they draw
- Stitched-together: understanding and generation modules bolted together with weak synergy
- Inputs: 3D scan datasets (ScanNet, Matterport3D, ARKitScenes) plus massive web video
- All assets unified into a canonical coordinate system
- Scene-level 3D oriented bounding boxes (OBB) extracted
- Frame-level object attributes distilled via projection, visibility filtering, and mask refinement
- Unified object-frame index: synced 3D/2D boxes, instance masks, partial point clouds, metric metadata
- Smaller vision encoder learning rate (5e-6 vs 5e-5) to preserve pretrained visual representations
- KL distillation applied only to general data, not spatial data: the base model's weak spatial ability means strong distillation would block new spatial knowledge
- Dynamic sequence packing: Flash Attention variable-length interface packs short sequences to cut padding waste
- General editing: open-source data + expert model distillation + text editing + multi-image editing
- Open-domain editing: natural scene changes
- Spatial editing: static-camera object transforms + dynamic-camera viewpoint transforms under unified spatial supervision
- Short: 1–2 sentences mimicking real user prompt distributions
- Long: paragraph-level—subject, objects, spatial relations, background, lighting, style, atmosphere
- Extended Long: textures, materials, spatial layout, subtle details
- Structured: JSON by dimension (subject, background, style, composition, visible text) for flexible data combination
- Long-tail distribution: 2% of categories hold 30%+ frequency, 50% of categories under 100K
- Tail fully kept; head inverse-logarithmically downsampled
- Weak-capability categories boosted +20~50% sampling
- Adaptive diversity sampling via taxonomy-tree level aggregation to avoid semantic redundancy
- Main + auxiliary sub-views, all oriented toward object centers
- Ray casting culls infeasible camera positions
- Dual prompt annotation: dense caption (natural language) + structured JSON (machine-parseable)
- Inference cost: 16B MMDiT + 8B MLLM is not cheap at inference time
- Reasoning tasks: on T2I-CoReBench's reasoning dimension, it still trails GPT Image 1—pure spatial enhancement helps abstract logical reasoning only so much
- Text-to-image version not yet released: currently only understanding and editing models are open; the core T2I model is "coming soon"
- Distilled version pending: a faster inference variant is still in the works
JoyAI-Image's premise is direct: understanding, generation, and editing are not three tasks but three facets of the same visual intelligence. Sharing one cognitive core lets understanding guide generation ("know the scene structure to draw it correctly") and generation reinforce understanding ("draw from another angle to verify spatial consistency").
This bidirectional loop is what "spatial intelligence" really means—not memorizing more geometry formulas, but building a persistent feedback loop between seeing and drawing.
2. Architecture: 8B Brain + 16B Paintbrush + VAE Bridge
| Component | Scale | Source | Role | |------|------|------|------| | MLLM | 8B | Qwen3-VL-8B-Instruct | Cognitive brain: understanding, parsing, instruction decomposition | | VAE | — | Wan-2.1-VAE | Compression bridge: pixels ↔ latent | | MMDiT | 16B | In-house | Generation engine: denoising, synthesis, editing |
Three-stage pipeline:
1. Understanding: MLLM parses user inputs (text/image/instructions) and extracts semantic-spatial cues 2. Encoding: VAE compresses images into latents, preserving high-frequency details (critical for small text rendering) 3. Generation: MMDiT's dual-stream architecture deeply fuses MLLM semantic conditioning with VAE visual features for iterative denoising
The MLLM is not confined to understanding tasks. During generation and editing, its last-layer hidden states serve as the MMDiT's primary conditioning signal—generation is guided by understanding throughout, not just pushed by text embeddings.
Training is progressive: fine-tune the MLLM's spatial understanding first, train MMDiT's generation from scratch, then jointly optimize editing precision.
3. How Spatial Intelligence Is "Awakened": The Data Engine
The core weapon is OpenSpatial, an automated data engine that synthesizes spatial QA pairs from 3D box-centric representations.
3.1 OpenSpatial's 3D Lifting
Traditional spatial data relies on expensive 3D scanning (LiDAR, multi-camera rigs). OpenSpatial instead lifts 3D structure from in-the-wild web video:
Key constraint: multi-view cycle consistency. A candidate 3D box is validated only when its projections align instance masks across multiple viewpoints—ensuring the 3D structure inferred from 2D video is not hallucinated.
3.2 Five Spatial Capabilities, 19 Subtasks
OpenSpatial-3M (3 million entries) covers:
| Capability | Example Subtasks | |------|-----------| | Spatial Measurement (SM) | Distance estimation, size comparison, depth estimation | | Spatial Relations (SR) | "between", "behind", relative position reasoning | | Camera Perception (CP) | Camera motion direction, focal length judgment | | Multi-view Consistency (MC) | Cross-view object correspondence, identity under viewpoint change | | Scene-Aware Reasoning (SAR) | 3D scene description, layout reasoning |
3.3 Training Data Mix (11.3M Total)
| Category | Count | Share | Purpose | |------|------|------|------| | General understanding | 6.1M | 54.25% | Preserve broad multimodal ability | | Spatial understanding | 3.4M | 29.65% | Core: OpenSpatial 3.3M + VST 49.4K | | Instruction enhancement | 1.4M | 11.98% | Rewrite low-entropy descriptions into dense instructions | | Spatial editing | 137.4K | 1.21% | Spatial instruction → visual transformation mapping | | Other | 328.1K | 2.89% | JD internal long-tail data |
The mix is deliberately uneven: spatial understanding is nearly 30%, but general understanding still exceeds half—preventing overfitting to spatial tasks at the cost of generalist capability.
3.4 Decoupled Learning Rates + Selective Distillation
4. Results: Spatial Understanding Matches Gemini-2.5-Pro, Long-Text Rendering SOTA
4.1 Spatial Understanding (13 Benchmarks)
| Model | Spa.Avg | Notes | |------|---------|------| | Gemini-2.5-Pro | 64.4 | Strongest closed-source | | JoyAI-Image-Und | 64.4 | Matches Gemini | | Qwen3-VL-8B (baseline) | 59.1 | +5.3 ↑ | | GPT-4o | 57.7 | Surpassed | | MiMo-VL-7B | 58.2 | Surpassed |
Notably, JoyAI-Image scores 60.1 on VSI-Bench (4D spatiotemporal reasoning) and 61.0 on AllAnglesBench (extreme viewpoint change)—the hardest 4D tasks. BLINK (low-level geometric cues) jumps from 49.5 to 61.0 (+11.5), showing the model genuinely learned to perceive depth and size. General benchmarks (MMBench, MMStar, OCRB) remain competitive.
4.2 Text-to-Image: LongText-Bench Dominance
| Model | LongText-EN | LongText-ZH | |------|-------------|-------------| | JoyAI-Image | 0.963 | 0.963 | | GPT Image 1 (High) | 0.956 | 0.619 | | Qwen-Image | 0.943 | 0.946 | | Z-Image | 0.935 | 0.936 | | Seedream 3.0 | 0.896 | 0.878 |
Long-text rendering is a major industrial pain point. JoyAI-Image hits 0.963 in both English and Chinese, while GPT Image 1 manages only 0.619 in Chinese—enabled by OCR-aware captioning and dedicated text-rendering SFT data. It is also SOTA on CVTG-2K (Word Accuracy 0.8739).
4.3 Composition and Reasoning (T2I-CoReBench)
| Model | Composition Mean | Reasoning Mean | Overall | |------|------------------|----------------|---------| | JoyAI-Image | 94.2 | 55.9 | 68.7 | | GPT Image 1 (High) | 79.8 | 69.0 | 72.6 | | Qwen-Image | 83.7 | 51.7 | 62.4 |
Composition (layout/structure/relations/text) crushes all rivals, confirming "understanding-driven generation" shines in complex compositional scenes. Reasoning (logic/counting/spatial) trails GPT Image 1 slightly but stays competitive.
4.4 Image Editing: +74.7% Spatial Editing Accuracy
On SpatialEdit-Bench, spatial editing accuracy improves 74.7% over the baseline. The editing data engine has three streams:
5. Key Engineering Details: Five Modules of the Data Pipeline
5.1 Five-Stage Progressive Filtering
| Stage | Resolution | Min Resolution | Aesthetic Score | IQA Retention | |------|--------|-----------|---------|----------| | Stage 1 | 208p | >128 | ≥3.0 | N/A | | Stage 2 | 512p | >256 | ≥4.6 | 34% | | Stage 3 | 1024p | >512 | ≥4.6 | 20% |
IQA uses a cascaded decision system: statistics (brightness/entropy/saturation/sharpness) + perceptual metrics (NIQE/CLIP-IQA/MUSIQ), verified at 90% human accuracy.
5.2 Four-Level Captioning
OCR-aware: an OCR-specific model extracts text tokens → fused with visual features → MLLM generates captions → post-filtering ensures full OCR coverage, content consistency, no language translation.
5.3 Rebalancing 285K Labels
5.4 Three-Dimension Human Scoring
| Dimension | Weight | 5-Point Standard | |------|------|------| | Aesthetics | 50% | Cinematic/artistic, harmonious light, composition, color | | Information density | 30% | Subject + objects + environment + interaction, rich texture | | Style purity | 20% | Unmistakable style, zero cross-style contamination, no "AI look" |
Quality control: 5% sentinel samples at a 90% accuracy threshold; 5% daily review, whole-batch rework if low-quality pass rate exceeds 5%.
5.5 Million-Scale Multi-View Generation (Blender)
6. Training Strategy: Four Stages from Pre-Training to RL
| Stage | Goal | Key Design | |------|------|------| | Pre-Training | Basic generation ability | Progressive resolution (208p→512p→1024p), Flow Matching | | Continue Training | Narrow distribution, raise quality | High-quality subset, reduced distribution entropy | | SFT | Complex text rendering + multi-view generation | Task-oriented small data, human annotation | | RL (Flow-GRPO) | Aesthetics + text alignment | Multi-reward models (aesthetics + text-image alignment), group-relative optimization |
7. Applications: A Foundation for VLA and World Models
The paper demonstrates two downstream capabilities:
1. Thinking with Novel Views: reasoning aided by generated novel viewpoints—e.g., "from this new angle, what was the occluded object?" 2. Reconstruction with Novel Views: 3D structure reconstruction from multi-views
These are exactly the capabilities VLA (Vision-Language-Action) models and world models need: robots must understand spatial relations to manipulate the physical world; world models must maintain cross-view consistency to predict future states. JoyAI-Image shows a unified vision model can provide this foundation.
8. Open-Source Contributions
| Resource | Status | Link | |------|------|------| | Paper | Released | arXiv:2605.04128 | | Code | Open-sourced | GitHub jd-opensource/JoyAI-Image | | Understanding model weights | Released | Hugging Face | | Editing model weights | Released | Hugging Face + ModelScope | | OpenSpatial-3M | Released | Hugging Face | | SpatialEdit dataset | Released | Hugging Face | | Diffusers support | Merged | 2026.05.08 | | ComfyUI support | Released | 2026.04.10 | | Distilled version | TBD | — | | Multi-image editing version | TBD | — | | Text-to-image version | TBD | — |
Open-sourcing models, data, training recipes, and tooling together is a rare level of investment from industry.
9. Why This Paper Matters
1. Unified architecture validated: MLLM+MMDiT achieves SOTA or near-SOTA across understanding, generation, and editing—not a stitched Frankenstein 2. An engineering path to spatial intelligence: OpenSpatial's 3D lifting + box-centric paradigm offers a scalable spatial data pipeline without expensive 3D scans 3. Mechanized bidirectional loop: understanding guides generation via conditioning signals; generation reinforces understanding via multi-view data 4. Industrial-grade data pipeline: five-stage filtering, four-level captions, 285K-label rebalancing, three-dimension human scoring—these details are what make reproduction possible 5. Groundwork for VLA and world models: spatial intelligence precedes embodied AI, and JoyAI-Image shows unified vision models can carry that load
10. Limitations
Paper: JoyAI-Image: Awaking Spatial Intelligence in Unified Multimodal Understanding and Generation. Lin Song et al., Joy Future Academy, JD. arXiv:2605.04128.
GitHub: https://github.com/jd-opensource/JoyAI-Image