SSD: Spatially Speculative Decoding Gives Autoregressive Image Generation Spatial Intuition
> "Nature is delightful in every domain." — Richard Feynman
---
🎨 Introduction: A Clumsy Painter
Imagine a painter creating a landscape. His painting order is bizarre: starting from the top-left corner, he strokes left to right, finishing one row before moving to the next. He never looks up at the whole picture, only at the area next to his brush tip—like a typewriter mechanically executing "paint the next pixel."
Stranger still, he has a strict rule: he must paint a pixel before deciding what the next one should be. He cannot infer upcoming content from the already-painted regions nearby.
You watch him work, frustrated: "The sky's gradient is right there in the painting! Just keep following that color downward! Why can't you look further ahead?"
This is how most autoregressive image generation models work. They forcibly "flatten" a 2D image into a 1D sequence and generate token by token, like writing prose—completely ignoring the most essential property of images: 2D spatial structure.
Until someone proposed a bold idea: what if the painter could look at the positions beside and below at the same time? What if he could "predict" the next few pixels in advance?
That is the story of SSD (Spatially Speculative Decoding).
---
🧩 Autoregressive Models: Kings of Language, Strangers to Images
What is autoregression?
"Autoregressive" sounds complex but is intuitive: predict the next one. Given "The weather today is very ___", your brain predicts likely next words. GPT works exactly this way, predicting one token at a time.
For language, this is natural—text is a 1D sequence.
Why aren't images 1D?
In a landscape photo, every pixel relates to its up, down, left, and right neighbors. The sky's color is continuous; the mountain ridgeline is continuous. This is 2D spatial locality—the essence of an image is two-dimensional.
Autoregressive models ignore this. They flatten images row by row into a long queue of tokens, like dismantling a beautiful puzzle and reciting "piece 1 is blue, piece 2 is blue, piece 3 is blue..."
The cost of flattening: the memory wall
A 1024x1024 image becomes 1,048,576 tokens. Generating it takes over a million steps—each producing one token while reprocessing the whole sequence. Worse: the model could use spatial information to infer multiple pixels at once, but doesn't. When it generates a "blue sky" token, the neighbor is almost certainly also blue sky. Yet traditional autoregressive generation must:
1. Generate token A (top-left) 2. Based on A, generate token B (to the right) 3. Based on A+B, generate token C (further right) 4. ...
Like someone who could take big strides but insists on shuffling an inch at a time. This is the memory wall: strict sequential dependencies prevent exploiting spatial correlation, severely limiting inference speed.
---
💡 SSD: Giving Models "Spatial Intuition"
SSD's core idea is surprisingly simple but strikingly effective: besides predicting the next token, the model also simultaneously predicts the horizontally adjacent token and the token directly below.
Why spatial speculation works
Like filling in a crossword: after writing "A", you can almost certainly guess the next letter is "P" (APPLE) without waiting for the whole word. SSD lets the model do the same:
- Primary prediction: generate the next token (traditional).
- Horizontal speculation: guess the right-adjacent token.
- Vertical speculation: guess the token directly below.
- Traditional objective: given the current token, predict the next token (1D direction).
- SSD objective: simultaneously predict the next token (horizontal), the lower-right token (diagonal), and the token below (vertical).
- Maximum speedup: 13.3x (in certain configurations)
- Typical speedup: 8-12x
- Key insight: the higher the resolution, the greater the speedup—high-resolution images have stronger spatial correlation (large regions share consistent colors), making speculative guesses more likely correct.
- Text: 1D sequence. Autoregression is natural.
- Images: 2D grids. Generation should be 2D (SSD's lesson).
- Video: 3D spatiotemporal volumes—future methods may explore "spatiotemporal speculative decoding."
- 3D scenes: point clouds, voxels, meshes, each with their own spatial relations.
- Molecules: graph structures requiring graph-level autoregression.
The model then verifies these guesses: correct guesses are accepted in batches; wrong guesses are rolled back and recomputed.
Key technique: spatially aligned prediction objectives
The key to SSD's success is aligning prediction targets with the image's natural geometry. During training, the model learns to predict not just "the next token in the sequence" but "spatially adjacent tokens" by modifying the training objective:
This training gives the model spatial thinking—no longer just a "sequence predictor" but a "spatial predictor."
The speculative decoding acceleration mechanism
Traditional autoregressive generation (slow): 1. Input: all tokens generated so far 2. Output: one token (the next) 3. Append and repeat 4. One million tokens = one million steps
SSD speculative decoding (fast): 1. Input: all generated tokens 2. Output: one primary token + multiple speculative tokens (horizontal, below) 3. A faster "verifier" checks the speculative tokens 4. If correct, multiple tokens are accepted at once 5. If wrong, roll back only to the error position 6. Spatial correlation in images is high, so acceptance rates are high—on average, multiple tokens are confirmed per step
Result: generating the same image takes only 1/13.3 of the original time.
---
📊 Experimental Results: Speed Soars, Quality Holds
SSD was validated on two industry-standard image generation benchmarks: DPG-Bench and GenEval.
Speedup
Quality preservation
Despite 10x+ speedups, image quality barely degrades. On DPG-Bench, SSD outputs match the baseline in visual quality, semantic consistency, and text-image alignment. On the stricter GenEval benchmark (complex prompt following), SSD maintains high quality as well.
Why quality doesn't drop
SSD doesn't "skip" computation—it "arranges" computation more sensibly. Traditional autoregressive models also "know" about spatial correlation but are trapped by the 1D framework. SSD simply lets the model naturally use the spatial knowledge it already learned. Like the painter: he isn't lazier, he's learned to watch several positions at once, finishing the same work with fewer motions.
---
🌐 Broader Implications: Respect the Natural Structure of Data
> Respecting the natural geometric structure of data can unlock huge computational efficiency.
Different modalities have different "shapes":
3D generation: if a model can simultaneously predict voxel neighbors along x, y, and z, 3D SSD may be possible.
Scientific computing: physics simulation, climate modeling, and materials science involve spatially structured data—SSD's idea could inspire new paradigms that speculate across both space and time.
---
🎯 Core Contributions
1. Identified the root problem: autoregressive image models flatten 2D images into 1D sequences, discarding spatial locality and causing severe computational bottlenecks. 2. Proposed the SSD framework: spatially speculative decoding via simultaneous horizontal and vertical neighbor prediction. 3. Achieved significant acceleration: up to 13.3x speedup with quality preserved. 4. Articulated a broader philosophy: matching generation strategies to data's natural geometry releases huge efficiency.
---
🌌 A Feynman-Style Reflection: Why Do Simple Things Get Overlooked?
Feynman said: "Knowing the name of something and understanding it are two different things."
Autoregressive image generation has existed for years. Why didn't anyone think of SSD earlier? Likely path dependence: NLP succeeded first with autoregressive models (GPT), and CV naturally "ported" the framework to images. Researchers optimized and scaled the framework, but few questioned whether it fit.
SSD reminds us: sometimes the deepest breakthrough is not adding complexity, but returning to the basic question—"why do we do it this way?" Why flatten images into 1D? Because text is 1D and text models succeeded. But images are not text. This simple fact was overlooked for years—until someone looked at a painting and said: "Wait—this picture is 2D. Let's think like a 2D being."
---
📚 References
1. Xiang, S., Zhang, Z., Yu, L., & Mao, C. "SSD: Spatially Speculative Decoding Accelerates Autoregressive Image Generation." arXiv:2606.20543, 2026. 2. Van Den Oord, A., et al. "Neural Discrete Representation Learning." NeurIPS, 2017. 3. Ramesh, A., et al. "Hierarchical Text-Conditional Image Generation with CLIP Latents." arXiv, 2022. 4. Yu, L., et al. "Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning." arXiv, 2023. 5. Leviathan, Y., et al. "Fast Inference from Transformers via Speculative Decoding." ICML, 2023. 6. Chen, C., et al. "PaGAL: A Prompt auto-Guided Self-Attentive Model for Language and Visual Tasks." CVPR, 2024.
*Daily paper recommendations | 2026-06-21* *"Even if the world forgets, I'll remember for you."*