FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents (arXiv, Apr 2025)
Overview
- Paper: FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents
- Authors: Nandan Thakur, Jimmy Lin, Sam Havens, Michael Carbin, Omar Khattab, Andrew Drozdov
- Source: https://arxiv.org/abs/2504.13128
- Date: April 2025
- Category: Evaluation of Search Engines
- Popular corpora are broad and generic rather than niche and domain-specific;
- Relevance signals from older benchmarks may leak into the training data of modern retrieval models;
- Offline metrics such as nDCG@10 and MRR increasingly diverge from real user satisfaction.
- Introduces a unified, reproducible framework for building retrieval benchmarks over technical documentation corpora;
- Emphasizes freshness and niche topics to mitigate data-contamination concerns when evaluating modern retrievers and RAG pipelines;
- Decomposes the standard pipeline into modular components — representations, retrievers, rerankers, generators, and feedback mechanisms — making it easier to evaluate each stage;
- Situates itself within the broader shift from static, single-shot retrieval metrics toward task-level evaluation (answer faithfulness, citation accuracy, multi-hop reasoning quality) in agentic and RAG systems;
- The forum note preserves the paper's evaluation design logic (datasets, metrics such as nDCG@10 / Recall@k, baselines like BM25 and dense retrieval, and ablations) while advising readers to verify quantitative results against the original PDF.
- ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation (arXiv:2311.09476)
- Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Articles (arXiv:2105.03011)
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv:2401.13178)
Motivation
Large-scale search, recommendation, and personalization systems have long faced challenges around efficiency, scalability, and understanding user intent. Traditional pipelines often treat retrieval, ranking, and generation as separate stages, which struggles to meet the needs of the LLM era: natural-language interaction, multi-hop reasoning, and access to fresh, specialized knowledge.
Existing IR benchmarks are increasingly problematic for evaluating retrieval-augmented generation (RAG) systems on technical content:
FreshStack addresses this by constructing fresh, realistic retrieval benchmarks on technical documents, using recent community-generated content so that evaluation is less contaminated and closer to real technical question-answering scenarios.
Key points
Engineering checklist (from the original analysis)
| Item | Question | Suggestion | |------|----------|------------| | Data | Does training/indexed data contain PII? How are versions managed? | Partitioned indexes, sanitization, rollback-capable embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascaded retrieval + early stopping, caching hot queries, async reranking | | Quality | Do offline gains translate to online satisfaction? | Interleaving experiments, human audits, citation checks | | Safety | Does open retrieval introduce poisoning/bias? | Source allowlists, adversarial detection, output filtering | | Cost | Per-query token and GPU cost? | Small-model routing, distillation, hybrid sparse+dense retrieval |
Takeaways
1. Architecture: cascaded retrieval + reranking + generation remains mainstream, but agentic paradigms are making retrieval frequency and strategy themselves learnable; 2. Data: high-quality instruction data and logs matter as much as model design; synthetic data requires care against leakage and distribution shift; 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation against human assessment — precisely the setting FreshStack targets; 4. Product: latency, cost, explainability, and safety are hard deployment constraints that cannot be optimized away on academic benchmarks alone.