Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini
> TL;DR: Google released Gemini Embedding 2, mapping text, images, audio, video, and arbitrary interleaved combinations into one unified vector space. It is not late-fusion (separate encoders then alignment) — it is native multimodal: one model directly ingests interleaved image-text, audio-video, and PDF documents, and outputs a single unified embedding. MSCOCO image-to-text R@1 78.2%, text-to-image 62.9%, Flickr30k image-to-text 97.3%, native audio retrieval 3.59 points above ASR pipelines, and SOTA on MMTEB multilingual and code tasks.
1. The Twilight of Late-Fusion
Existing multimodal embedding models (CLIP, ALIGN, SigLIP 2, CoCa) all follow a late-fusion approach:
- Text goes through a text encoder, images through an image encoder, audio through an audio encoder
- Each is encoded separately, then aligned into a shared space
- Result: they handle single modalities and paired data (image-text pairs), but not truly interleaved inputs
- An image with a caption, asking "at which second does the red object appear in the image"
- A video plus a spoken user question, asking "what did the person in the video say"
- A PDF document (text + charts + screenshots), asking "sum the sales in the table on page three"
- A podcast audio clip, asking "what topic are the guests discussing"
- Pure text (up to 8192 tokens)
- Images (up to 6 per request, PNG/JPEG)
- Video (up to 120 seconds, MP4/MOV, sampled at 1 FPS up to 32 frames)
- Audio (native processing, no ASR transcription needed)
- Documents (PDF, up to 6 pages)
- Any interleaving: image+text, audio+text, video with image-text prompts...
- Stage 1: Initialize from Gemini, leveraging its existing multimodal knowledge
- Stage 2: Task-specific, modality-specific training
- Stage 3: Cross-modality training
- End-to-end throughout, with all modalities optimized under one loss
- Multilingual tasks: 69.9 (SOTA)
- Code tasks: 84.0 (SOTA)
- Same-language retrieval (PassageInLang): 75.58 vs 73.58 (+2.0)
- Cross-language retrieval (PassageCrossLang): 72.56 vs 67.55 (+5.01)
- MicroVQA (microscopy image QA)
- ArtCap (art captioning)
- AstroLLaVA (astronomical images)
- Recipe1M (cooking recipes)
- Storage-sensitive scenarios use 768 dims (75% space savings)
- Precision-sensitive scenarios use 3072 dims
- Same model, no retraining
- MSR-VTT: 63.3% → 76.1% (+12.8)
- Vatex: 68.8% → 79.5% (+10.7)
- Document retrieval: PDFs (text + charts + screenshots) embedded as one unit, no OCR + image encoding + text encoding pipeline
- Video recommendation: video content embedded directly, no frame extraction + tagging + captioning
- Audio search: podcasts, meeting recordings, support calls embedded directly, no ASR
- Interleaved multimodal retrieval: "find the second at which the image with the red car appears in the video" — joint image-text prompt locates the timestamp
- Paper: Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini
- arXiv: 2605.27295
- Authors: Madhuri Shanbhogue, Zhe Li, Shanfeng Zhang, et al. (Gemini Embedding Team, Google)
- Institution: Google DeepMind
- Date: 2026-05-26
- API: https://deepmind.google/models/gemini/embedding/
- Official blog: https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-embedding-2/
- Supported modalities: Text (8192 tokens), Image (6 images), Video (120 seconds, up to 32 frames at 1 FPS), Audio (native), Document (PDF, 6 pages)
- Output dimensions: 3072 (default), shrinkable to 1536/768 (MRL)
- Core mechanism: Gemini initialization, multi-task multi-stage contrastive learning, native interleaved input processing
- Key numbers: MSCOCO image→text R@1 78.2%, Flickr30k image→text 97.3%, MMTEB multilingual 69.9, code 84.0, native audio mrr@10 73.99 (vs ASR 70.40), MSR-VTT fine-tuned 76.1%
What are "interleaved inputs"?
Late-fusion models are helpless here. They either need a complex pipeline (ASR transcription → text embedding + image embedding → fusion) or simply fail.
Gemini Embedding 2's answer: not multiple encoders — one encoder.
2. Native Multimodal: One Model Ingests Everything
2.1 Architecture
Initialized from Gemini, using a Transformer with bidirectional attention. Inputs can be arbitrary combinations of modalities:
Processing pipeline: 1. Raw inputs → Gemini's native tokenizer (each modality converted to tokens) 2. Unified token sequence → bidirectional-attention Transformer (initialized from Gemini) 3. Mean pooling → linear projection to target dimension 4. Output: 3072-dimensional vector (default), shrinkable to 1536/768 via Matryoshka Representation Learning
Key point: all modalities are mixed at the token level, not encoded separately and then concatenated. The model can see in attention layers that "this word sits next to that image" or "this audio segment corresponds to this text."
2.2 Training
Large-scale multi-task, multi-stage contrastive learning:
Contrastive objective: semantically similar content (regardless of modality) sits close together in vector space; unrelated content sits far apart.
3. Performance: Comprehensive Cross-Modal SOTA
3.1 Cross-Modal Retrieval
| Task | Benchmark | Metric | Gemini Embedding 2 | Best Comparison | |------|------|------|-------------------|-----------------| | Image→Text | MSCOCO | R@1 | 78.2% | 72.0% (SigLIP 2) | | Text→Image | MSCOCO | R@1 | 62.9% | 56.0% (SigLIP 2) | | Image→Text | Flickr30k | R@1 | 97.3% | 95.0% (SigLIP 2) | | Text→Image | Flickr30k | R@1 | 89.0% | 84.0% (SigLIP 2) | | Image→Image | ImageNet | R@1 | 83.5% | 79.0% (CLIP) | | Text→Video | MSR-VTT | N@10 | 63.3% | 57.9% (Amazon Nova) | | Text→Video | Vatex | N@10 | 68.8% | 55.2% (Amazon Nova) |
On video retrieval, Gemini Embedding 2 clearly outpaces Amazon Nova (AWS's SOTA): MSR-VTT 63.3% vs 57.9%, Vatex 68.8% vs 55.2%.
3.2 Text Embeddings: MMTEB
On MMTEB (Massive Multilingual Text Embedding Benchmark):
It beats all existing models on the leaderboard. Code retrieval is a particularly notable leap — multimodal embedding models have historically been weak on code.
3.3 Native Audio: Crushing ASR Pipelines
Traditional audio retrieval: audio → ASR → text → text embedding. Two steps, with error propagation.
Gemini Embedding 2 processes raw audio directly, no transcription.
MSEB (Multilingual Speech Embedding Benchmark) results:
| Setting | mrr@10 | |------|--------| | ASR pipeline | 70.40 | | Native audio | 73.99 |
A 3.59-point gain. Breaking it down:
Cross-language gains are larger. ASR error propagation is worse in cross-lingual scenarios — one misrecognized word gets amplified when translated. Native audio bypasses this bottleneck.
3.4 Zero-Shot Generalization: From Astronomy to Cooking
The paper tests several specialized domains:
All zero-shot, no domain-specific fine-tuning. Results generally beat CLIP variants, showing the embeddings are general enough to avoid per-domain training.
4. Engineering Details That Make It Usable
4.1 Matryoshka Representation Learning
Default output is 3072 dimensions, but can dynamically shrink to 1536 or 768 without cliff-edge performance loss:
4.2 Efficient Fine-Tuning: Thousands of Steps + O(k) Data
With in-domain data, just a few thousand steps and few samples yield significant gains. Paper examples:
Important for enterprise: the general model is strong, but adding your own data makes it stronger at controlled cost.
4.3 Model Souping
Weights from multiple task-specific fine-tuned models are averaged (model soup), balancing specialized knowledge and generalization. No ensemble inference (multiple models voting) — just average the weights into one model. Inference cost unchanged, results better than any single fine-tuned model.
5. Infrastructure for Agentic RAG
The next step beyond RAG is Agentic RAG — agents retrieving not just documents but images, video, audio, and code. Gemini Embedding 2 provides unified infrastructure:
Google's official use-case demo shows retrieving a specific timed event in video via a joint image-text prompt. Under traditional architectures this needs at least 3 models (image, text, video encoders) plus complex cross-modal alignment logic. Gemini Embedding 2 does it with one model.