LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

Source: https://arxiv.org/abs/2308.14508 (arXiv, August 2023)

Authors: Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, et al. (13 authors in total).

Overview

LongBench is a bilingual (English and Chinese), multitask benchmark for measuring how well large language models understand long contexts. It was introduced to address a gap in LLM-era evaluation: classic search and recommendation pipelines split retrieval, ranking, and generation into separate stages, whereas modern LLM systems must process very long inputs end to end, requiring multi-hop reasoning, natural language interaction, and integration with external knowledge.

Key points

• Bilingual and multitask: the benchmark spans task categories such as long-document question answering, summarization, and other context-heavy workloads, in both English and Chinese.
• Unified evaluation framework: it consolidates previously scattered long-context evaluation work into a comparable, reproducible protocol, lowering the entry cost for follow-up research.
• Component-level clarity: the framework makes explicit how representation, retrieval, reranking, generation, and feedback mechanisms factor into long-context performance, aiding engineering adoption.
• Interfaces to emerging paradigms: the work is positioned alongside LLM tool calling, reinforcement learning, and multi-agent collaboration, outlining paths from research prototypes to industrial systems.
• Open problems: evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual/multimodal extension remain open challenges.

Method framing

The evaluation follows a typical pipeline: problem formalization → model/system design → construction or training → inference pipeline. Inputs (queries, documents, user context) are encoded as dense or sparse representations or structured prompts; core modules may include retrievers, rerankers, planners, memory modules, and tool interfaces; learning strategies include supervised fine-tuning, contrastive learning, distillation, and reinforcement learning; inference strategies include single-pass and iterative retrieval, parallel sub-queries, early stopping, and budget control.

Experimental context

Relevant datasets and metrics in this evaluation space include MS MARCO, BEIR, Natural Questions, and domain corpora, with metrics such as nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost. Baselines typically span BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, and commercial search APIs.

> Note: specific numerical results should be verified against the original paper's tables. This report summarizes the evaluation design based on the abstract and public metadata.

Insights for search, recommendation, and personalization

1. Architecture: cascaded retrieval + reranking + generation remains mainstream, but agentic paradigms increasingly treat "whether and how often to retrieve" itself as a learnable decision. 2. Data: high-quality instruction data and click/session logs are equally critical; synthetic data requires safeguards against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation. 4. Product: latency, cost, explainability, and safety are hard constraints for industrial deployment — do not optimize academic benchmarks alone.

Limitations and future work

Possible limitations include experiment scale bounded by GPU budgets, benchmark–real-user distribution mismatch, English-centric data leaving cross-lingual generalization uncertain, and safety risks of agent systems on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommender systems.

Related entries

• Evaluation of Retrieval-Augmented Generation: A Survey, May 2024, arXiv
• ARES: An Automated Evaluation Framework for RAG, 2023, arXiv
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents, 2024, arXiv
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts, 2021, arXiv

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool invocation | | Gen-IR | Generative Information Retrieval |