> Paper: Text-to-Image Models Need Less from Text Encoders Than You Think > Authors: Nurit Spingarn*, Noa Cohen*, Tamar Rott Shaham, Tomer Michaeli (*equal contribution) > Affiliations: Technion – Israel Institute of Technology & MIT CSAIL > arXiv: 2606.03715 | Submitted: June 2, 2026 > Project page: https://nsping13.github.io/contextless-TTI/
Text-to-image research has long assumed that stronger text encoders (CLIP → T5-XXL → Qwen3) yield better generation. This study dismantles that assumption with an elegant set of experiments.
Core finding: DiT-based text-to-image models extract only two minimal pieces of information from text encoders — (1) which tokens belong to the same word, and (2) the word's position in the sentence. All complex semantic reasoning — compositionality, attribute binding, spatial relations — is done by the image model itself.
The authors' Bag-of-Position-Tagged-Words (BoPTW) embedding, which strips all syntax, semantic dependencies, and cross-word context, achieves ~65% non-inferiority on SD3, FLUX.1, and FLUX.2 — remarkably close to the 70–90% of full embeddings. The conclusion: make the text encoder lighter, make the image model stronger.
Background
| Stage | Models | Text encoder | Scale | |-------|--------|--------------|-------| | Early | SD 1.x/2.x | CLIP ViT-L/14 | ~124M | | Mid | Imagen, SD3, FLUX.1 | T5-XXL + CLIP | ~11B+ | | Recent | FLUX.2 | Qwen3 (LLM) | ~30B+ |
Prior clues: Wang et al. (CVPR 2025) showed T5-base can match T5-XXL on image quality via visual distillation; CLIP's bag-of-words behavior and insensitivity to word order were long known. But whether DiT models actually use contextual information from encoders had not been systematically tested.
Method: Three Contextless Embeddings
1. Bag-of-Tokens (BoT) — each token's embedding averaged over many unrelated sentences, erasing all context, order, and position. Ambiguous for multi-token words (~32% of MS-COCO words are split by T5). 2. Bag-of-Words (BoW) — adds the knowledge of which tokens form one word, but no position. 3. BoPTW — adds absolute word position; each word is averaged only from sentences where it appears in the same position. The key construction: if BoPTW works, the image model must be inferring semantic relations itself.
Built from CC3M and MS-COCO 2017 embeddings (≥10 contexts per token, supplemented with Claude-generated sentences). Evaluation: Gemma-3 three-way blind VLM comparison, non-inferiority framework, CLIP score, FID/KID on DrawBench, GenEval, and MS-COCO 2014.
Results
| Embedding | Non-inferiority rate | |-----------|---------------------| | BoT | >40% | | BoW | >50% | | BoPTW | ≥65% (vs 70–90% full) |
By category (BoPTW): single objects 88–100%; colors/counts/spatial 60–80%; text rendering only 27–37%.
The decisive DiT vs UNet comparison:
| Model | Architecture | BoPTW non-inferiority (DrawBench) | |-------|-------------|-----------------------------------| | SD 2.1 | UNet | 0.2% | | SDXL | UNet | 4% | | SD3 / FLUX.1 / FLUX.2 | DiT | ~65% |
UNet models produce nothing meaningful from BoPTW — the semantic reconstruction ability comes from the Transformer architecture itself, not the encoder.
Analysis & Implications
- Redefined division of labor: encoders supply "word + position"; DiT reconstructs semantics via self-attention. "Generation is understanding."
- Text-side inference cost could drop an order of magnitude with a lightweight word-embedding + positional module.
- Strategic resource shift: invest on the image side; connect to Wang et al.'s 50× T5 compression.
- Open question: would a model trained from scratch on BoPTW alone work equally well, or are big encoders needed during training but disposable at inference?
Limitations
1. Text rendering still needs full encoding (27–37%) — encoder value lies in precise character alignment, not semantic understanding. 2. Failure patterns unclear (negation, comparatives, nested modifiers?). 3. Generality across architectures (PixArt, Hunyuan-DiT) unverified. 4. Training vs inference distinction unresolved; multilingual and idiomatic phrases untested.
Conclusion
A counterintuitive lesson: more complex ≠ better. Years of scaling text encoders were largely wasted effort for DiT models, which perform language understanding themselves. Future architectures may favor lightweight text encoders and stronger image models, letting generation itself carry understanding.
References
1. Spingarn, N., et al. (2026). Text-to-Image Models Need Less from Text Encoders Than You Think. arXiv:2606.03715. 2. Wang, L., et al. (2025). Scaling Down Text Encoders of Text-to-Image Diffusion Models. CVPR 2025. 3. Esser, P., et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (SD3). ICML 2024. 4. Rombach, R., et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022. 5. Saharia, C., et al. (2022). Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (Imagen). NeurIPS 2022. 6. Radford, A., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision (CLIP). ICML 2021.