ColPali: Efficient Document Retrieval with Vision Language Models
Paper: https://arxiv.org/abs/2407.01449
Authors: Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, et al. (7 authors)
Published: 2024-06-27
Original Abstract
> Documents are visually rich structures that convey information through text, but also figures, page layouts, tables, or even fonts. Since modern retrieval systems mainly rely on the textual information they extract from document pages to index documents -often through lengthy and brittle processes-, they struggle to exploit key visual cues efficiently. This limits their capabilities in many practical document retrieval applications such as Retrieval Augmented Generation (RAG). To benchmark current systems on visually rich document retrieval, we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of various page-level retrieval tasks spanning multiple domains, languages, and practical settings. The inherent complexity and performance shortcomings of modern systems motivate a new concept; doing document retrieval by directly embedding the images of the document pages. We release ColPali, a Vision Language Model trained to produce high-quality multi-vector embeddings from images of document pages. Combined with a late interaction matching mechanism, ColPali largely outperforms modern document retrieval pipelines while being drastically simpler, faster and end-to-end trainable. We release models, data, code and benchmarks under open licenses at https://hf.co/vidore.
Key Points
- Problem: Modern document retrieval pipelines depend on OCR/text extraction, which is lengthy, brittle, and discards visual cues (figures, tables, page layouts, fonts) that matter in real documents.
- Benchmark: The authors introduce ViDoRe (Visual Document Retrieval Benchmark), a collection of page-level retrieval tasks across multiple domains, languages, and practical settings for comparing document retrieval systems.
- Approach: ColPali is a Vision Language Model trained to produce high-quality multi-vector embeddings directly from page images — no OCR pipeline required.
- Matching: Embeddings are consumed via a late interaction mechanism, combining the effectiveness of multi-vector matching with precomputable document representations.
- Results: ColPali largely outperforms modern document retrieval pipelines while being drastically simpler, faster, and end-to-end trainable.
- Open release: Models, data, code, and benchmarks are available under open licenses at https://hf.co/vidore.
- Late interaction multi-vector approaches trade index size for retrieval quality; budget storage and ANN infrastructure accordingly.
- Evaluate latency (p99), embedding versioning, and index update workflows before production deployment.
- Validate that offline retrieval gains translate into end-to-end answer quality, using citation checks and human audits alongside nDCG-style metrics.
- Hybrid-Vector Retrieval for Visually Rich Documents (arXiv:2510.22215)
- EA-VTR: Event-Aware Video-Text Retrieval (ECCV 2024)
- Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions
- Original paper: ColPali: Efficient Document Retrieval with Vision Language Models. arXiv:2407.01449
Why It Matters for RAG and Search
For Retrieval-Augmented Generation (RAG), the quality of retrieval over visually rich documents (PDFs, scanned reports, slides) is often the bottleneck. ColPali reframes indexing as a visual embedding problem, eliminating fragile preprocessing stages and enabling end-to-end optimization.