jina-vlm: A 2.4B-Parameter Multilingual Vision-Language Model with Attention Pooling

Forum topic · ✨步子哥 · 2025-12-23

Summary

jina-vlm is a 2.4B-parameter open-source vision-language model (VLM) designed to address two common weaknesses of small VLMs: multilingual degradation after visual adaptation and high inference cost. It combines a SigLIP2-So400M/14-384 vision encoder with a Qwen3-1.7B-Base language decoder, connected by an attention-pooling connector. Images of arbitrary resolution are handled via overlapping tiling (12 tiles plus a global thumbnail, 112-pixel overlap, 266-pixel stride), while features from two ViT layers (third and ninth from the last) are concatenated, 2x2 attention-pooled to cut visual tokens by 4x (from 9,477 to 2,366 tokens), and projected via SwiGLU. Training uses two-stage full-parameter tuning: alignment with captioning data plus 15% pure text, then instruction fine-tuning on VQA/OCR/math mixes, totaling 5M multimodal samples and 12B multilingual text tokens across 30+ languages. The model achieves a 72.3 average on English VQA benchmarks, beats comparable 2B models on multilingual benchmarks (MMMB 78.8, Multilingual MMBench 74.3), attains the best hallucination robustness (POPE 90.3), and retains most text-only capabilities despite slight MMLU-Pro decline. This report summarizes the architecture, training recipe, and evaluation results.

jina-vlm: A Small Multilingual Vision-Language Model (2.4B)

jina-vlm is a 2.4B-parameter open-source multilingual vision-language model (VLM) built to tackle two persistent problems in small VLMs:

1. Multilingual degradation — many VLMs lose language ability after visual adaptation, excelling on English benchmarks but stumbling in other languages. 2. Resource cost — high-performing models are often too expensive for researchers and practitioners.

jina-vlm addresses both by pairing a SigLIP2 vision encoder with a Qwen3 language backbone, connected via attention pooling, achieving state-of-the-art multilingual VQA among ~2B open VLMs (MMMB, Multilingual MMBench) and a 72.3 average on English VQA benchmarks.

Architecture

Vision encoder: SigLIP2-So400M/14-384, a 27-layer ViT processing 378×378 inputs into a 27×27 grid of 14×14 patches.
Arbitrary-resolution handling: overlapping tiling — a default of 12 tiles plus one global thumbnail, each tile overlapping by 112 pixels with a 266-pixel stride. Larger images are intelligently downsampled; tile count can increase at inference with linear memory growth.
Connector: features are taken not from the final ViT layer but concatenated from the 3rd-last (layer 24) and 9th-last (layer 18) layers, capturing both low-level spatial detail and high-level semantics:

\mathbf{H}_{\text{concat}} = [\mathbf{H}^{(-3)}; \mathbf{H}^{(-9)}] \in \mathbb{R}^{N \times 2d_v}

2×2 attention pooling: neighborhood means serve as queries, compressing tokens 4× while preserving spatial structure:

\mathbf{q}_i = \frac{1}{4} \sum_{j \in N_i} \mathbf{h}_j, \quad \mathbf{Q} = [\mathbf{q}_1; \dots; \mathbf{q}_M] \in \mathbb{R}^{M \times 2d_v}

\mathbf{H}_{\text{pooled}} = (\text{softmax} \left( \frac{\mathbf{Q}\mathbf{W}_Q(\mathbf{H}_{\text{concat}}\mathbf{W}_K)^{\top}}{\sqrt{d_k}} \right) \mathbf{H}_{\text{concat}}\mathbf{W}_V)^{\top} \mathbf{W}_O \in \mathbb{R}^{M \times d_v}

Projection: a SwiGLU layer maps pooled features into the language model dimension:

\mathbf{H}_{\text{proj}} = (\text{Swish}(\mathbf{H}_{\text{pooled}}\mathbf{W}_1) \odot (\mathbf{H}_{\text{pooled}}\mathbf{W}_2)) \mathbf{W}_3 \in \mathbb{R}^{M \times d_l}

Language decoder: Qwen3-1.7B-Base with special tokens (<im_start>, <im_end>, <im_col>); input/output embeddings are not shared.

Efficiency of attention pooling

Per the paper's Table 1: tokens drop from 9,477 (no pooling) to 2,366 (pooled); LLM prefill FLOPs fall from 27.2 to 6.9 TFLOPs; KV cache memory drops from 2.12 GB to 0.53 GB — an overall 4× reduction.

Training

Two-stage full-parameter training:

1. Alignment: captioning data (PixmoCap, PangeaIns) covering natural scenes, documents, and charts, plus 15% pure-text data (PleiAS/common corpus) to prevent text degradation. Higher connector learning rate (2e-4 vs. ViT 6e-6, LLM 2e-5) for fast adaptation. 2. Instruction fine-tuning: VQA and reasoning mixtures (LLaVA OneVision, Cauldron, etc.), 30K single-source steps then 30K mixed-batch steps.

Total: 5M multimodal samples + 12B multilingual text tokens (30+ languages, ~half English).

| Hyperparameter | Pretraining | Fine-tuning | |--------|--------|------| | Warmup ViT | 10% | 10% | | Warmup Connector | 1% | 10% | | Warmup LLM | 10% | 10% | | LR ViT | 6e-6 | 5e-6 | | LR Connector | 2e-4 | 5e-6 | | LR LLM | 2e-5 | 1e-5 | | Cosine decay | 0.1 | 0.1 | | Betas | 0.9, 0.95 | 0.9, 0.95 | | Batch size | 128 | 256 | | Steps | 25K | 60K | | Samples | 3.2M | 15.3M | | Tokens | 10B | 37B | | GPU hours | 296 | 1,000 |

Text capability is largely retained, with only MMLU-Pro declining (46.4 → 30.3), attributed to instructions favoring concise visual responses.

Evaluation

English VQA (Table 3)

| Model | AI2D | ChartQA | TextVQA | DocVQA | InfoVQA | OCRBench | SEED-2 Plus | CharXiv (RQ/DQ) | Overall | |------|------|---------|---------|--------|---------|----------|-------------|-----------------|------| | jina-vlm | 82.0 | 81.9 | 83.2 | 90.6 | 71.6 | 778 | 67.2 | 32.3/63.5 | 72.3 | | Qwen2-VL-2B | 74.7 | 73.5 | 79.7 | 89.2* | 64.0* | 809 | 62.4 | 23.3/55.0* | 66.4 | | Qwen3-VL-2B | 76.9 | 77.2 | 79.5 | 92.3* | 71.9* | 858 | 67.3* | 28.8/62.3 | 71.6 | | InternVL3-2B | 78.6 | 80.2 | 77.0 | 87.4* | 67.1* | 835 | 64.6 | 28.3/54.7 | 69.2 | | InternVL3.5-2B | 78.8 | 80.7 | 76.5 | 88.5* | 69.3* | 836 | 68.0 | 31.6/65.0 | 71.6 |

*Computed with VLMEvalKit. OCRBench divided by 10 for percentage alignment.*

jina-vlm also scores MME 1965.8 and RealWorldQA 68.2 (overall multimodal 67.4, real-world 61.9).

Multi-image and hallucination (Table 5)

| Model | BLINK | MuirBench | MMT-Bench | Overall (MI) | HallBench | POPE | Overall (Hall) | |-------|-------|-----------|-----------|-----------|--------|------|-------------| | jina-vlm | 50.1 | 34.7 | 57.2 | 47.3 | 39.1 | 90.3 | 64.7 | | Qwen2-VL-2B | 44.4 | 25.5* | 55.1 | 41.7 | 41.7 | 87.9* | 64.8 | | Qwen3-VL-2B | 53.8 | 47.4 | 60.0* | 53.7 | 44.5 | 88.9* | 66.7 | | InternVL3-2B | 50.3 | 38.8 | 59.5 | 49.5 | 42.5 | 89.6 | 66.1 | | InternVL3.5-2B | 51.3 | 44.0 | 58.5 | 51.3 | 48.6 | 87.2 | 67.9 |

Multi-image reasoning (47.3) is limited by training data, but hallucination control is best-in-class (POPE 90.3).

Mathematical reasoning (Table 6)

| Model | MMMU | MathVista | MathVision | MathVerse | WeMath | LogicVista | Overall | |------|------|-----------|------------|-----------|--------|------------|------| | jina-vlm | 45.6 | 59.5 | 19.2 | 23.9 | 17.1 | 33.3 | 33.1 | | Qwen2-VL-2B | 41.1 | 43.0 | 12.4 | 17.3* | 10.9* | 27.3* | 25.3 | | Qwen3-VL-2B | 53.4 | 61.3 | 31.6 | 22.7* | 28.0* | 35.4* | 38.7 | | InternVL3-2B | 48.6 | 57.0 | 21.7 | 25.3 | 22.4 | 36.9 | 35.3 | | InternVL3.5-2B | 59.0 | 71.8/61.5† | 42.8/26.5† | 53.4/35.3† | 48.5/19.1† | 47.7/41.4† | 50.7 |

Overall math score 33.1, comparable to InternVL3-2B.

Text-only benchmarks (Table 7)

| Model | MMLU | MMLU-Pro | GSM-8k | ARC-C | HellaSwag | Overall | |------|------|----------|--------|-------|-----------|------| | jina-vlm | 56.1 | 30.3 | 71.3 | 77.3 | 59.4 | 58.9 | | Qwen3-1.7B | 62.6 | 46.4 | 75.3 | 73.4 | 59.0 | 63.3 |

Multilingual understanding

Best-in-class at 2B scale: MMMB 78.8 and Multilingual MMBench 74.3, plus strong MTVQA performance — evidence that explicitly balancing 30+ languages during training prevents the usual English-centric degradation.

Limitations and outlook

Multi-tile processing is computationally expensive at high resolution; tiling may fragment global spatial structure, affecting counting or cross-boundary reasoning (the global thumbnail mitigates but does not eliminate this).
Safety alignment and multi-image training remain insufficient.

Future work points toward more efficient native-resolution processing, spatial-task optimization, and knowledge transfer at larger scales. Overall, jina-vlm demonstrates that small VLMs can balance efficiency and capability — a democratizing step for resource-constrained AI.

References

1. Koukounas, A., et al. (2025). JINA-VLM: Small Multilingual Vision Language Model. arXiv:2512.04032v2. 2. Tschannen, M., et al. (2025). SigLIP2: Efficient Vision Encoder for VLMs. 3. Yang, A., et al. (2025). Qwen3: Advanced Language Backbone for Multimodal Tasks. 4. Deitke, M., et al. (2025). PixmoCap: Diverse Caption Dataset for VLM Training. 5. Yue, X., et al. (2025). PangeaIns: Multilingual Instruction Data for Vision-Language Models.