TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants (SIGIR 2024)
This forum post indexes the SIGIR 2024 resource paper describing the TREC iKAT 2023 test collection, designed for evaluating Conversational and Interactive Knowledge Assistants.
Key points
- What it is: A shared benchmark (test collection) released through TREC for evaluating systems that assist users in conversational, interactive information seeking.
- Task focus: Multi-turn dialogue search, where assistants must resolve context-dependent user needs, maintain consistency across turns, and retrieve relevant evidence from a document collection.
- Personalization: Includes personal textual profiles capturing user background and preferences, requiring systems to tailor responses accordingly.
- Evaluation targets: Response relevance, faithfulness/attribution of generated answers, and appropriate use of user context — moving beyond static ranking metrics toward task-level and human-centered assessment.
- Provides a reproducible reference point for conversational search systems in the LLM era, complementing static benchmarks (MS MARCO, BEIR, Natural Questions).
- Supports end-to-end evaluation of pipelines combining retrieval, reranking, planning, and generation.
- Useful for studying process-oriented metrics such as task success, citation accuracy, and multi-turn coherence.
- Paper page: https://dl.acm.org/doi/abs/10.1145/3626772.3657860
- Venue: SIGIR 2024 (resource track)
- Evaluation of Retrieval-Augmented Generation: A Survey
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Articles
- ARES: An Automated Evaluation Framework for RAG
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents