CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning

Forum topic · 小凯 · 2026-04-06

Summary

CoME-VL (arXiv:2604.03231) is a research paper on vision-language modeling that proposes a modular fusion framework combining complementary visual encoders. While most VLMs rely on a single CLIP-style contrastive vision encoder optimized for cross-modal alignment and retrieval, self-supervised encoders like DINO capture richer dense semantics and greater robustness on recognition tasks. CoME-VL integrates a contrastively trained vision encoder with a self-supervised DINO encoder, performing representation-level fusion via entropy-guided multi-layer aggregation with orthogonality constraints. Experiments across multiple vision-language benchmarks show CoME-VL consistently outperforms single-encoder baselines, with average improvements of 4.9% on visual understanding tasks and 5.4% on grounding tasks. The work, by Ankan Deria, Komal Kumar, Xilin He, and colleagues, demonstrates that scaling the fusion of complementary visual representations is an effective direction for stronger vision-language models.

Paper Overview

Field: Computer Vision (CV) Authors: Ankan Deria, Komal Kumar, Xilin He, et al. Published: 2026-04-03 arXiv: 2604.03231

Abstract

Recent vision-language models (VLMs) typically rely on a single vision encoder trained with contrastive image-text objectives, such as CLIP-style pretraining. While contrastive encoders are effective for cross-modal alignment and retrieval, self-supervised visual encoders often capture richer dense semantics and exhibit stronger robustness on recognition and understanding tasks.

This work investigates how to scale the fusion of these complementary visual representations for vision-language modeling. The authors propose CoME-VL: Complementary Multi-Encoder Vision-Language, a modular fusion framework that integrates a contrastively trained vision encoder with a self-supervised DINO encoder.

The approach performs representation-level fusion by:

Entropy-guided multi-layer aggregation
Orthogonality constraints to keep the fused representations complementary (abstract truncated in source)

Key Results

Extensive experiments across multiple vision-language benchmarks show that CoME-VL consistently outperforms single-encoder baselines:

+4.9% average improvement on visual understanding tasks
+5.4% improvement on grounding tasks

Takeaway

Combining contrastive encoders (strong at cross-modal alignment and retrieval) with self-supervised encoders (strong at dense semantics and robustness) through a modular, representation-level fusion framework yields meaningful gains for vision-language models.

--- *Auto-collected on 2026-04-06*