No Hard Negatives Required: Concept-Centric Learning Improves Compositionality Without Hurting Zero-Shot Capabilities

Forum topic · 小凯 · 2026-03-28

Summary

Researchers from Samsung Labs propose a concept-centric learning approach for contrastive vision-language (V&L) models that improves compositional reasoning without requiring hard negative samples. The authors identify two root causes limiting compositionality: long training captions do not force compositional representations, and final global pooling in text and image encoders discards information needed for binding concepts. Their solutions are: (1) using standard NLP tools to extract short, concept-centric caption fragments aligned with images, and (2) introducing a parameter-free cross-modal attention pooling to obtain concept-centric visual embeddings. Combined with a simple auxiliary contrastive loss, these changes achieve state-of-the-art performance on standard compositionality benchmarks while maintaining or improving zero-shot classification and retrieval performance, with no added inference cost. The paper (arXiv 2603.25722) contrasts with prior methods that generate custom hard negative training data, which are benchmark-specific, fail to generalize, and often degrade core V&L capabilities. Code is available at https://github.com/SamsungLabs/concept_centric_clip.

Paper Overview

Field: Computer Vision Authors: Hai X. Pham, David T. Hoffmann, Ricardo Guerrero, Brais Martinez Published: 2026-03-26 arXiv: 2603.25722

Abstract (Original)

Contrastive vision-language (V&L) models remain a popular choice for various applications. However, several limitations have emerged, most notably the limited ability of V&L models to learn compositional representations. Prior methods often addressed this limitation by generating custom training data to obtain hard negative samples. Hard negatives have been shown to improve performance on compositionality tasks, but are often specific to a single benchmark, do not generalize, and can cause substantial degradation of basic V&L capabilities such as zero-shot or retrieval performance, rendering them impractical. In this work we follow a different approach. We identify two root causes that limit compositionality performance of V&Ls:

1. Long training captions do not require a compositional representation; 2. Final global pooling in both text and image encoders leads to the complete loss of information necessary for learning binding.

As remedies, we propose two simple solutions:

1. We use standard NLP software to obtain short, concept-centric caption fragments and align them with images. 2. We introduce a parameter-free cross-modal attention pooling that takes concept-centric visual embeddings from the image encoder.

With these changes and a simple auxiliary contrastive loss, we achieve state-of-the-art performance on standard compositionality benchmarks while maintaining or even improving strong zero-shot and retrieval capabilities. There is no additional inference cost.

Summary

Hard negative-based approaches improve compositionality benchmarks but are dataset-specific, fail to generalize, and degrade zero-shot and retrieval performance.
The paper identifies two root causes of poor compositionality: long captions don't enforce compositional learning, and global pooling destroys binding-relevant information.
Proposed fixes: concept-centric short caption fragments for training, plus parameter-free cross-modal attention pooling for image embeddings.
Results: SOTA on standard compositionality benchmarks with no loss (and sometimes gains) in zero-shot and retrieval performance, and no extra inference cost.

Code

https://github.com/SamsungLabs/concept_centric_clip

No Hard Negatives Required: Concept-Centric Learning Improves Compositionality Without Hurting Zero-Shot Capabilities

Summary

Paper Overview

Abstract (Original)

Summary

Code

Tags