StyleVAR: A New Take on Style Transfer Using Autoregressive Models

An Old Problem, A New Approach

You take a landscape photo with a good composition but flat colors, then look at Van Gogh's *Starry Night* and wish your photo had those swirling brushstrokes. This is the classic scenario of style transfer: preserving the content structure of one image while applying the artistic style of another.

The problem is not new. Since Gatys et al.'s classic 2015 paper, style transfer has evolved through nearly a decade: from optimization-based methods (hundreds of iterations per image), to feed-forward networks (milliseconds per image), to diffusion models (higher quality but slow). Researchers have long balanced quality against speed.

In April 2026, Duke University researchers Liqi Jing, Dingming Zhang, Peinian Li, and Lichen Zhu proposed a new idea: formulate style transfer as conditional discrete sequence modeling, solved with a visual autoregressive model (VAR).

The paper is titled *StyleVAR: Controllable Image Style Transfer via Visual Autoregressive Modeling*.

Why Autoregressive Models?

Autoregressive models generate one element at a time—like writing an essay, where each character depends on everything written before. In vision, the VAR (Visual Autoregressive Modeling) framework decomposes an image into a multi-scale token sequence and generates it coarse-to-fine with a Transformer: outline first, details later.

StyleVAR's core insight: style transfer can be viewed as conditional generation—not generating from scratch, but generating under the joint constraints of "content" and "style."

Blended Cross-Attention: Letting Content and Style "Negotiate"

Imagine a painter consulting two references simultaneously—a photo (content) and a style painting (style)—negotiating between them rather than copying either. StyleVAR's Blended Cross-Attention is the mathematical realization of this negotiation:

1. Target representations (the tokens being generated) serve as Key and Value, maintaining the generation history 2. Style features and content features serve as Queries, deciding "what to extract from the history" 3. A scale-dependent blending coefficient controls the relative influence of style versus content at each stage

The key is "scale-dependent": at coarse stages (low resolution), content should dominate to guarantee correct composition; at fine stages (high resolution), style should dominate, since brushstrokes, textures, and color character live mainly in details.

Two-Stage Training: Supervision First, Reinforcement Later

• Stage 1: Supervised Fine-Tuning (SFT). Fine-tuning from a pretrained VAR checkpoint on large numbers of content–style–target image triplets to learn basic style transfer ability.
• Stage 2: Reinforcement Learning (GRPO). Using Group Relative Policy Optimization—a DeepSeek-proposed RL algorithm that needs no separate reward model, estimating policy gradients via within-group relative comparison—with a DreamSim perceptual reward as the objective.

Because VAR's multi-scale tokens contribute unequally to final quality (coarse tokens affect global structure, fine tokens affect local detail), a uniform reward could over-optimize some scales while neglecting others. StyleVAR introduces per-action normalization weighting so every scale receives fair optimization.

Results

Across three benchmarks covering in-domain, near-domain, and out-of-domain distributions, StyleVAR outperforms the AdaIN baseline on nearly all metrics:

• Style Loss: lower
• Content Loss: lower
• LPIPS: better
• SSIM: better
• DreamSim (perceptual reward): better
• CLIP Similarity: better

The GRPO stage brings further gains over SFT, especially on reward-aligned perceptual metrics. Qualitatively, StyleVAR excels on landscapes and architecture—transferring texture style well while preserving semantic structure.

Known Limitations

The paper honestly notes several shortcomings:

• Poor face style transfer. Faces have long been difficult because the boundary between "content" and "style" is far blurrier than in landscapes—changing style easily changes identity.
• Generalization gap on internet imagery. Performance drops on out-of-distribution images, suggesting the model's understanding of style is bounded by training data coverage.
• Insufficient content diversity. Limited content types in training data constrain handling of novel content.

Honest Assessment

StyleVAR's contribution is a new paradigm: reframing style transfer from an "image transformation" problem into a "conditional sequence generation" problem. Benefits include:

1. Theoretical elegance—no bespoke architecture needed; it leverages VAR's generative capability directly 2. Better extensibility—it can inherit improvements from the broader VAR ecosystem 3. Quality assurance—autoregressive models have consistently shown strong generation quality

Caveats remain: autoregressive inference speed is a bottleneck (token-by-token generation won't beat single-pass feed-forward methods, though the paper doesn't report detailed timing), and two-stage training adds complexity whose cost depends on quality requirements.

Overall, StyleVAR is a meaningful, if not revolutionary, innovation—pointing to a promising direction in which future image editing tools might genuinely be built on autoregressive models.

---

Paper: arXiv:2604.21052 Code: GitHub - StyleVAR Model weights: HuggingFace - StyleVAR