How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Clinical RAG System for Breast Lesion Literature

Forum topic · 小凯 · 2026-10-25

Summary

A 2026 arXiv paper (2610.31606) presents an evidence-aware retrieval evaluation protocol for a clinical RAG system built over breast lesion management literature. Using an 18-document corpus and an 18-query development benchmark, the authors combined dense embeddings, BM25, and cross-encoder reranking, testing chunk sizes from 128 to 1024 tokens, overlap rates from 0 to 20%, and four retriever configurations. The selected hybrid retriever, with 384-token chunks and 20% overlap, retrieved accepted evidence within the top three results for 17 of 18 queries, achieving Recall@3, R-Precision, MRR@3, and nDCG@3 of 0.94 each. Notably, ground-truth curation expanded identified evidence spans from 25 to 37 without changing retrieval rankings, demonstrating that relevance annotations meaningfully influence reported evidence coverage while rankings remain stable under cross-encoder reranking. The study highlights annotation sensitivity in retrieval evaluation and provides a practical protocol for evidence-aware clinical RAG assessment.

Paper Overview

Field: NLP
Authors: Miguel Mochizuki Silva, Samuel Ismael, Mateus C. Dias, Thaís Gaudêncio, Yuri Malheiros, Barufaldi
Published: 2026-10-22
arXiv: 2610.31606

Abstract

Purpose: To describe an evidence-aware retrieval evaluation protocol for a clinical RAG system over breast lesion management literature.

Methods: An 18-document corpus and 18-query development benchmark were used. Retrieval combined dense embeddings, BM25, and cross-encoder reranking. Chunk sizes of 128 to 1024 tokens, overlaps of 0 to 20%, and four retriever configurations were evaluated.

Results: The selected hybrid retriever, using 384-token chunks and 20% overlap, retrieved accepted evidence in the top three results for 17 of 18 queries. Recall@3, R-Precision, MRR@3, and nDCG@3 each reached 0.94. Ground-truth curation increased evidence spans from 25 to 37 without changing retrieval rankings.

Conclusion: Cross-encoder reranking achieved the highest retrieval scores. Expanded relevance annotations increased the amount of identified evidence without changing rankings, highlighting the sensitivity of retrieval evaluation to ground-truth curation.

Key Takeaways

Hybrid retrieval (dense embeddings + BM25 + cross-encoder reranking) is effective for clinical evidence retrieval.
384-token chunks with 20% overlap performed best in this benchmark.
Ground-truth annotation choices affect reported evidence coverage (25 → 37 spans) even when rankings remain stable.