PDF Retrieval with Vision Language Models: ColPali Document Search with Vespa

*Source: Vespa blog*

Overview

This post originates from a Vespa engineering blog article on PDF retrieval with vision language models, centered on ColPali and how it can be used for document search from Vespa. ColPali represents PDF pages using vision language model embeddings, allowing retrieval to operate directly on page images instead of depending on OCR and layout-parsing pipelines that often degrade on tables, figures, and complex layouts.

> Original abstract excerpt: "PDF Retrieval with Vision Language Models, about ColPali and using it for document search from Vespa."

Key points

• Motivation: Traditional PDF search pipelines (OCR → text extraction → indexing) lose visual structure such as tables, charts, and multi-column layouts. ColPali instead embeds page images directly with a vision language model.
• Method shape: ColPali follows the late interaction paradigm — page images are encoded into multi-vector representations that interact with query token embeddings at scoring time, balancing accuracy and precomputability.
• System integration: Vespa provides indexing and serving support for ColPali-style multi-vector representations, enabling large-scale document search over page-level embeddings.
• Positioning in IR: The work fits into the evolution from BM25 → dense bi-encoders → cross-encoders → late interaction (e.g., ColBERT-style) → generative/agentic retrieval, addressing the efficiency–effectiveness–maintainability trade-off triangle.

System pipeline

A typical pipeline implied by this approach follows four steps:

1. Input & representation: PDF pages are rendered as images and encoded into dense multi-vector representations; queries are encoded by the same or a paired model. 2. Core modules: Retrieval over page-level embeddings, with late-interaction scoring for re-ranking or direct matching. 3. Serving strategy: Vespa handles multi-vector storage, approximate nearest-neighbor retrieval, and scoring at scale. 4. Evaluation: Standard IR metrics (e.g., nDCG@10, Recall@k) applied against text-pipeline baselines such as OCR + BM25 or dense retrieval.

Engineering trade-offs

• Accuracy vs. latency: Late interaction is more expressive than single-vector bi-encoders but requires more compute at query time.
• Indexing cost: Encoding every page image is more expensive than text extraction, but avoids fragile parsing steps.
• Robustness: Visual embeddings capture layout and figures that OCR pipelines miss, improving retrieval on visually complex documents.

Caveats

This entry is derived from the post's metadata and abstract-level information; specific benchmark numbers, model configurations, and Vespa implementation details should be verified against the original Vespa blog post.

Glossary

| Term | Meaning | |------|---------| | ColPali | A vision-language-model-based document retrieval method using page image embeddings and late interaction | | Late interaction | Scoring paradigm where query and document token embeddings interact at query time while document representations remain precomputable | | RAG | Retrieval-Augmented Generation | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric |

Related entries

• Evaluating search relevance part 2 – Phi-3 as relevance judge
• Increase web search accuracy and efficiency with dynamic filtering
• Investigating ChatGPT Search: Insights from 80 Million Clickstream Records
• A Coding Implementation to Build a Conversational Research Assistant