STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases (arXiv, Apr 2024)
Paper: https://arxiv.org/abs/2404.13207 Authors: Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, et al. (10 authors total) Category: Evaluation of Search Engines / Information Retrieval
Overview
STaRK is a benchmark for evaluating large language model (LLM)-based retrieval over semi-structured knowledge bases. Its queries are natural-language user requests that blend:
- Precise relational constraints (e.g., graph-structured relationships between entities), and
- Fuzzy textual requirements (e.g., semantic descriptions of attributes).
- Provides a unified framework for comparing retrieval methods that mix textual and relational reasoning.
- Decomposes the system stack into retrievers, rerankers, planners, and generators, easing engineering adoption.
- Offers reproducible benchmark datasets and evaluation protocols, lowering the entry cost for follow-up research.
- Interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration.
- Highlights open problems: evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual/multimodal extension.
- Metrics: Hit@k, Recall@k, MRR, nDCG@10, plus latency and token cost.
- Baselines: BM25, dense retrieval, cross-encoder reranking, no-retrieval LLMs.
- Caveat: Exact numerical results should be verified against the original PDF; offline metric gains do not always translate into online user satisfaction, and LLM-as-judge results should be cross-validated with human evaluation.
- Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
- ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts (arXiv:2105.03011)
The benchmark is built on three real-world semi-structured knowledge bases spanning Amazon products, Yelp businesses, and Harvard courses, making it relevant to search, recommendation, and personalization applications.
Motivation
Traditional retrieval pipelines separate retrieval, ranking, and generation, which struggles to meet LLM-era demands for natural-language interaction, multi-hop reasoning, and up-to-date knowledge. STaRK targets the core question: can retrieval systems jointly exploit both the textual properties of nodes and the relational structure of a knowledge graph, rather than treating documents as independent text?
Key Points
Evaluation Considerations
Typical evaluation setups in this space include:
Insights for Search / Rec / Personalization
1. Architecture: Cascaded retrieve-rerank-generate remains dominant, but agentic paradigms are making retrieval strategy itself learnable. 2. Data: High-quality instruction data and session logs matter; synthetic data risks leakage and distribution shift. 3. Product: Latency, cost, explainability, and safety are hard constraints in production, not just academic benchmarks.
Limitations and Future Work
Potential limitations include benchmark–real-user distribution mismatch, English-centric data, and safety risks of agentic retrieval on the open web. Future directions include better test-time compute allocation, deeper integration with knowledge graphs, and causal/fairness constraints for recommendation.