BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval

Authors: Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, et al. (15 authors in total) Source: https://arxiv.org/abs/2407.12883 (July 2024, arXiv) Category: Evaluation of Search Engines / Information Retrieval

One-line summary

BRIGHT is a new benchmark of real-world queries with human-annotated relevance judgments designed to test retrieval systems on tasks where finding the right documents requires substantial reasoning, not just keyword or semantic matching.

Background and Motivation

Most retrieval benchmarks reward lexical or shallow semantic matching. However, many real-world information needs—debugging code, solving math problems, answering scientific research questions—require retrieving documents whose surface forms differ greatly from the query and whose relevance can only be established through multi-step reasoning. Existing retrievers, even strong ones evaluated on BEIR, have not been systematically tested on such reasoning-intensive scenarios. BRIGHT was proposed to fill this gap.

Key Contributions

• A realistic, reasoning-intensive benchmark: 1,104 queries collected from natural contexts, each with human-annotated relevance judgments.
• Diverse domains: queries span coding, math, data science, leetcode-style problems, and research disciplines including biology, economics, psychology, and climate science.
• Extensive evaluation of existing methods: BM25, dense retrievers (including models fine-tuned on MS MARCO), and LLM-based approaches are benchmarked.
• Reasoning-enhanced strategies: the authors explore query rewriting with large language models and leveraging GPT-4-style reasoning to improve retrieval, showing gains but also revealing a large remaining gap.

Key Findings

• Existing retrieval methods—including BM25 and state-of-the-art dense retrievers—perform substantially worse on BRIGHT than on conventional benchmarks like BEIR, exposing a significant weakness in reasoning-intensive retrieval.
• Simple LLM-based tricks (query expansion, reasoning over candidate documents) help, but they do not close the gap.
• The results suggest that retrieval research should shift focus from pure semantic similarity toward models that can reason about document relevance.

Significance

BRIGHT complements prior evaluation efforts for retrieval-augmented generation (e.g., RAG evaluation surveys, ARES, QASA-style domain QA datasets) by providing a stress test for the retrieval component itself. It is positioned as a long-lasting testbed to motivate retrieval systems capable of multi-hop, domain-specific reasoning, which is increasingly important for LLM-driven search and agentic systems.

Limitations and Future Directions

• The benchmark is English-centric; cross-lingual generalization remains unexplored.
• Relevance judgments are costly, limiting scale compared to synthetic benchmarks.
• Future work includes retrieval models that integrate explicit reasoning, better query decomposition, and tighter coupling with downstream generation.

References

• Original paper: BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval

Related Entries

• Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
• ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Articles (arXiv:2105.03011)
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv:2401.13178)