CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning

Forum topic · 小凯 · 2026-04-06

Summary

CoME-VL (arXiv:2604.03231) is a vision-language framework that fuses complementary visual representations from two encoders instead of relying on a single CLIP-style contrastive encoder. While contrastive encoders excel at cross-modal alignment and retrieval, self-supervised encoders such as DINO capture richer dense semantics and show stronger robustness on recognition and understanding tasks. CoME-VL is a modular fusion framework that integrates a contrastively trained vision encoder with a self-supervised DINO encoder, performing representation-level fusion via entropy-guided multi-layer aggregation with orthogonal regularization. Experiments across multiple vision-language benchmarks show that CoME-VL consistently outperforms single-encoder baselines, with an average improvement of 4.9% on visual understanding tasks and 5.4% on grounding tasks, demonstrating that scaling complementary multi-encoder fusion is an effective direction for vision-language modeling.

Paper Overview

Research area: Computer Vision (CV)
Authors: Ankan Deria, Komal Kumar, Xilin He, et al.
Release date: 2026-04-03
arXiv: 2604.03231

Abstract

Recent vision-language models (VLMs) typically rely on a single vision encoder trained with contrastive image-text objectives, such as CLIP-style pretraining. While contrastive encoders are effective for cross-modal alignment and retrieval, self-supervised visual encoders often capture richer dense semantics and exhibit stronger robustness on recognition and understanding tasks.

In this work, the authors investigate how to scale the fusion of these complementary visual representations for vision-language modeling. They propose CoME-VL: Complementary Multi-Encoder Vision-Language, a modular fusion framework that integrates a contrastively trained vision encoder with a self-supervised DINO encoder.

The approach performs representation-level fusion by:

1. Entropy-guided multi-layer aggregation with orthogonal regularization 2. (further fusion mechanisms described in the full paper)

Results

Extensive experiments across multiple vision-language benchmarks show that CoME-VL consistently outperforms single-encoder baselines:

+4.9% average improvement on visual understanding tasks
+5.4% improvement on grounding tasks

These results suggest that combining contrastive and self-supervised representations through modular multi-encoder fusion is a promising scaling direction for vision-language models.

---

*Auto-collected on 2026-04-06.*