FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents (arXiv, Apr 2025)

Overview

This post summarizes an April 2025 arXiv paper proposing FreshStack, a framework for building realistic benchmarks to evaluate retrieval over technical documents.

• Paper: FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents
• Authors / Affiliations: Nandan Thakur, Jimmy Lin, Sam Havens, Michael Carbin, Omar Khattab, Andrew Drozdov
• Link: <https://arxiv.org/abs/2504.13128>
• Type: Academic paper (Evaluation of Search Engines)

Key points

• Motivation: Traditional retrieval pipelines separate retrieval, ranking, and generation, which struggles to meet LLM-era demands for natural language interaction, multi-hop reasoning, and up-to-date knowledge. Existing retrieval benchmarks do not reflect the realities of searching technical documentation, motivating FreshStack.
• Positioning: The work sits at the intersection of information retrieval and large-scale search/recommendation systems, asking how the responsibilities of retrieval, ranking, generation, and tool calling should be re-divided in the LLM era — where inference budget and action space (whether to retrieve, how many times, which tools) become first-class design variables.
• Typical pipeline framing (as described in the post):
1. Input & representation: encode queries, documents, and user context into dense/sparse representations or structured prompts; 2. Core modules: retriever, reranker, planner, memory, tool interfaces; 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, RL (including process rewards), synthetic data bootstrapping; 4. Inference strategies: single-shot retrieval, iterative retrieval, parallel sub-queries, early stopping and budget control.

Evaluation context

The post reviews the typical experimental setup for retrieval research, against which FreshStack-style benchmarks should be read:

• Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora;
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost;
• Baselines: BM25, dense retrieval, cross-encoder reranking, no-retrieval LLMs, commercial search APIs.

The post emphasizes that as RAG and agentic search evolve, evaluation is shifting from static nDCG toward process-level metrics such as task success rate, citation accuracy, and completeness of multi-hop reasoning chains.

Insights for search / recommendation systems

1. Architecture: Cascaded retrieve–rerank–generate remains mainstream, but agentic paradigms increasingly treat retrieval count and policy as learnable objects. 2. Data: High-quality instruction data and click/session logs remain critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation. 4. Production: Latency, cost, explainability, and safety are hard constraints for industrial deployment — academic benchmarks alone are not enough.

Engineering checklist (from the post)

| Area | Question | Recommendation | |------|----------|----------------| | Data | Does training/indexed data contain PII? How are versions managed? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascade + early stop, cache popular queries, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation checks | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Small-model routing, distillation, hybrid sparse+dense |

Limitations and related work

The post notes common limitations in this research area: experiment scale constrained by GPU budgets, mismatch between benchmarks and real user distributions, English-centric data with unknown cross-lingual generalization, and safety risks of agentic systems on the open web.

Cross-references from the original post

• AI Search Has A Citation Problem (CJR, Mar 2025)
• Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts (arXiv:2105.03011)
• ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
• Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks (arXiv:2505.24876)
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv:2401.13178)

Glossary

• IR — Information Retrieval
• RAG — Retrieval-Augmented Generation
• LTR — Learning to Rank
• nDCG — Normalized Discounted Cumulative Gain
• Agentic Search — modeling search as sequential decision-making and tool calling
• Gen-IR — Generative Information Retrieval

> Editor's note: The original forum post is largely a template-style analysis built from the paper's abstract and public metadata; the paper's abstract text was not reproduced in the source. Quantitative claims should be verified against the full PDF before citation.