No Hard Negatives Required: Concept-Centric Learning Improves Compositionality Without Hurting Zero-Shot Performance

Forum topic · 小凯 · 2026-03-28

Summary

Researchers from Samsung Labs propose a concept-centric learning approach for contrastive vision-language (V&L) models that improves compositional understanding without the drawbacks of hard-negative training. The paper identifies two root causes limiting compositionality: long training captions that do not require compositional representations, and final global pooling in text and image encoders that discards information needed for binding concepts. The proposed fixes are: (1) extracting short concept-centric caption fragments using standard NLP tools and aligning them with images, and (2) introducing a parameter-free cross-modal attention pooling that produces concept-centric visual embeddings from the image encoder. Combined with a simple auxiliary contrastive loss, the method achieves state-of-the-art results on standard compositionality benchmarks while preserving or improving zero-shot and retrieval performance, with no added inference cost. Code is available at https://github.com/SamsungLabs/concept_centric_clip. The paper is available on arXiv: 2603.25722.

Paper Overview

Field: Computer Vision Authors: Hai X. Pham, David T. Hoffmann, Ricardo Guerrero, Brais Martinez Published: 2026-03-26 arXiv: 2603.25722 Code: https://github.com/SamsungLabs/concept_centric_clip

Abstract

Contrastive vision-language (V&L) models remain a popular choice for various applications. However, several limitations have emerged, most notably the limited ability of V&L models to learn compositional representations. Prior methods often addressed this limitation by generating custom training data to obtain hard negative samples. Hard negatives have been shown to improve performance on compositionality tasks, but are often specific to a single benchmark, do not generalize, and can cause substantial degradation of basic V&L capabilities such as zero-shot or retrieval performance, rendering them impractical.

Key Contributions

The authors identify two root causes limiting compositionality performance of V&L models:

1. Long training captions do not require a compositional representation. 2. Final global pooling in both the text and image encoders causes the information necessary for learning binding to be completely lost.

Proposed Solutions

1. Use standard NLP software to obtain short concept-centric caption fragments and align them with images. 2. Introduce a parameter-free cross-modal attention pooling to obtain concept-centric visual embeddings from the image encoder.

Results

With these changes and a simple auxiliary contrastive loss, the method achieves state-of-the-art performance on standard compositionality benchmarks while maintaining or improving strong zero-shot and retrieval capabilities — with no additional inference cost.

--- *Auto-collected on 2026-03-28*