TL;DR: Google released Gemini Embedding 2, a native multimodal embedding model that maps text, images, audio, video, PDFs, and arbitrary interleaved combinations into a single vector space. It is not late-fusion (separate encoders then alignment) — one model directly consumes mixed image-text inputs, interleaved audio-video, and PDF documents and outputs one unified embedding. Key results: MSCOCO image-to-text R@1 78.2%, text-to-image 62.9%, Flickr30k image-to-text 97.3%, native audio retrieval 3.59 points above ASR pipelines, and SOTA on MMTEB multilingual and code tasks.
1. The Twilight of Late-Fusion
Existing multimodal embedding models (CLIP, ALIGN, SigLIP 2, CoCa) follow the late-fusion route:
- Text goes to a text encoder, images to an image encoder, audio to an audio encoder
- Each is encoded separately, then aligned into a unified space
- Result: they handle single modalities and paired data (image-text pairs), but not truly interleaved inputs
- An image with a text caption, asking "at which second does the red object appear"
- A video plus a spoken user query, asking "what did this person say in the video"
- A PDF document (text + charts + screenshots), asking "sum the sales figures in the table on page 3"
- A podcast audio, asking "what topic are the guests discussing"
- Pure text (up to 8192 tokens)
- Images (up to 6 per request, PNG/JPEG)
- Video (up to 120 seconds, MP4/MOV, sampled at 1 FPS, max 32 frames)
- Audio (processed natively, no ASR transcription needed)
- Documents (PDF, up to 6 pages)
- Any interleaving: image+text, audio+text, video with an image-text prompt...
- Stage 1: Initialize from Gemini, leveraging its existing multimodal knowledge
- Stage 2: Task-specific, modality-specific training
- Stage 3: Cross-modality training
- The whole pipeline is end-to-end, with all modalities optimized under one loss
- Multilingual tasks: 69.9 (SOTA)
- Code tasks: 84.0 (SOTA)
- Same-language retrieval (PassageInLang): 75.58 vs 73.58 (+2.0)
- Cross-language retrieval (PassageCrossLang): 72.56 vs 67.55 (+5.01)
- MicroVQA (microscopy image QA)
- ArtCap (art captioning)
- AstroLLaVA (astronomy images)
- Recipe1M (cooking recipes)
- Storage-sensitive scenarios use 768 dims (75% space savings)
- Accuracy-sensitive scenarios use 3072 dims
- Same model, no retraining needed
- MSR-VTT: 63.3% → 76.1% (+12.8)
- Vatex: 68.8% → 79.5% (+10.7)
- Document retrieval: PDFs (text + charts + screenshots) embedded in one pass — no OCR + image encoding + text encoding pipeline
- Video recommendation: video content embedded directly — no frame extraction + tagging + text description
- Audio search: podcasts, meeting recordings, support calls embedded directly — no ASR
- Interleaved multimodal retrieval: "find at which second the image of the red car appears in the video" — a joint image+text prompt locates the video timestamp
- Paper: Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini
- arXiv: 2605.27295
- Authors: Madhuri Shanbhogue, Zhe Li, Shanfeng Zhang, et al. (Gemini Embedding Team, Google)
- Institution: Google DeepMind
- Date: 2026-05-26
- API: https://deepmind.google/models/gemini/embedding/
- Official blog: https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-embedding-2/
- Supported modalities: Text (8192 tokens), Image (6 images), Video (120 seconds, 1 FPS, max 32 frames), Audio (native), Document (PDF, 6 pages)
- Output dimensions: 3072 (default), scalable to 1536/768 (MRL)
- Core mechanisms: initialization from Gemini, multi-task multi-stage contrastive learning, native interleaved input processing
- Key numbers: MSCOCO image→text R@1 78.2%, Flickr30k image→text 97.3%, MMTEB multilingual 69.9, code 84.0, native audio mrr@10 73.99 (vs ASR 70.40), MSR-VTT fine-tuned 76.1%
Examples of interleaved inputs:
Late-fusion models are helpless on these tasks. They either need complex pipelines (ASR transcription → text embedding + image embedding → fusion) or simply cannot do it.
Gemini Embedding 2's solution: not multiple encoders — one encoder.
2. Native Multimodality: One Model Consumes Everything
2.1 Architecture
Initialized from Gemini, using a bidirectional-attention Transformer. Inputs can be any combination of modalities:
Processing pipeline: 1. Raw input → Gemini native tokenizer (each modality converted to tokens) 2. Unified token sequence → bidirectional-attention Transformer (initialized from Gemini) 3. Mean pooling over the sequence → linear projection to target dimension 4. Output: 3072-dimensional vector (default), scalable to 1536/768 (Matryoshka Representation Learning)
Key point: all modalities are mixed at the token level — not encoded separately and then concatenated. The model sees, in the attention layers, "this word sits next to this image" and "this audio corresponds to this text."
2.2 Training
Large-scale multi-task, multi-stage contrastive learning:
The contrastive objective: semantically similar content (regardless of modality) lands close in vector space; unrelated content lands far apart.
3. Performance: SOTA Across the Board
3.1 Cross-Modal Retrieval
| Task | Benchmark | Metric | Gemini Embedding 2 | Best comparison model | |------|-----------|--------|--------------------|-----------------------| | Image→Text | MSCOCO | R@1 | 78.2% | 72.0% (SigLIP 2) | | Text→Image | MSCOCO | R@1 | 62.9% | 56.0% (SigLIP 2) | | Image→Text | Flickr30k | R@1 | 97.3% | 95.0% (SigLIP 2) | | Text→Image | Flickr30k | R@1 | 89.0% | 84.0% (SigLIP 2) | | Image→Image | ImageNet | R@1 | 83.5% | 79.0% (CLIP) | | Text→Video | MSR-VTT | N@10 | 63.3% | 57.9% (Amazon Nova) | | Text→Video | Vatex | N@10 | 68.8% | 55.2% (Amazon Nova) |
On video retrieval, Gemini Embedding 2 clearly outpaces Amazon Nova (AWS's SOTA): MSR-VTT 63.3% vs 57.9%, Vatex 68.8% vs 55.2%.
3.2 Text Embeddings: MMTEB
On the Massive Multilingual Text Embedding Benchmark (MMTEB):
It exceeds every existing model on the leaderboard. Code retrieval is a particularly notable leap — multimodal embedding models have historically been weak on code.
3.3 Native Audio: Beating ASR Pipelines
Traditional audio retrieval: audio → ASR → text → text embedding. Two steps, with error propagation.
Gemini Embedding 2 processes raw audio directly, with no transcription.
MSEB (Multilingual Speech Embedding Benchmark) results:
| Setup | mrr@10 | |-------|--------| | ASR pipeline | 70.40 | | Native audio | 73.99 |
A gain of 3.59 points. Breakdown:
The cross-lingual gain is larger. ASR error propagation is worse in cross-lingual settings — a misrecognized word gets amplified when translated into another language. Native audio bypasses this bottleneck.
3.4 Zero-Shot Generalization: From Astronomy to Cooking
The paper tests several specialized domains:
All zero-shot, with no domain-specific fine-tuning. Results generally beat CLIP variants, demonstrating the embeddings' generality without per-domain training.
4. Engineering Details That Make It Usable
4.1 Matryoshka Representation Learning
Default output is 3072 dimensions, but can be dynamically scaled to 1536 or 768 without cliff-edge performance loss. This means:
4.2 Efficient Fine-Tuning: Thousands of Steps + O(k) Data
Fine-tuning on in-domain data requires only a few thousand steps and small sample counts for significant gains. Examples from the paper:
This matters for enterprises — the general model is strong, but adding your own data makes it stronger at controllable cost.
4.3 Model Souping
Weights from multiple task-specific fine-tuned models are averaged (model soup) to balance specialization and generalization. No ensemble inference (multiple models voting) — instead, averaged weights become a single model for inference. Inference cost is unchanged while performance beats any single fine-tuned model.
5. Infrastructure for Agentic RAG
The next step for RAG (Retrieval-Augmented Generation) is Agentic RAG — agents that retrieve not only documents but also images, video, audio, and code snippets. Gemini Embedding 2 provides the unified infrastructure:
One official Google use case demonstrates retrieving a specific timed event in a video using a joint image-text prompt. Under traditional architectures this requires at least 3 models (image, text, video encoders) plus complex cross-modal alignment logic. Gemini Embedding 2 solves it with one model.