FaithDial: A Faithful Benchmark for Information-Seeking Dialogue (Dec 2022, MIT Press)
Metadata
| Field | Content | |-------|---------| | Title | FaithDial: A Faithful Benchmark for Information-Seeking Dialogue | | Venue | Transactions of the Association for Computational Linguistics (TACL), December 2022, MIT Press | | Source | https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00529/114373/FaithDial-A-Faithful-Benchmark-for-Information | | Resource type | Academic paper | | Section | Evaluation of Search Engines |
One-sentence summary
FaithDial is a benchmark for evaluating the faithfulness of responses in information-seeking dialogue, measuring whether generated replies are supported by the underlying knowledge source rather than hallucinated.
Background and motivation
In large-scale search, recommendation, and personalization systems, information retrieval has long faced challenges around efficiency, scalability, and understanding user intent. Traditional pipeline approaches often treat retrieval, ranking, and generation as separate stages, which makes it hard to meet the demands of the LLM era for natural-language interaction, multi-hop reasoning, and up-to-date knowledge.
FaithDial addresses a specific and important sub-problem in this space: in knowledge-grounded dialogue, models frequently produce fluent responses that are not supported by the provided evidence. The benchmark is designed to make such failures measurable, enabling systematic evaluation of dialogue systems on faithfulness in open-domain information-seeking settings.
Core contributions
- A faithfulness benchmark for information-seeking dialogue that evaluates responses against a knowledge source.
- A controlled evaluation setting that goes beyond fluency-based metrics, targeting hallucination and unsupported claims in grounded conversations.
- A reusable reference point for subsequent work on grounded dialogue, retrieval-augmented generation, and hallucination evaluation.
- Reliability of automatic faithfulness evaluation vs. human judgment;
- Latency and cost constraints in grounded dialogue systems;
- Hallucination and safety in open-ended generation;
- Cross-lingual and multimodal extensions.
- AI Search Has A Citation Problem, Mar 2025, CJR
- Evaluation of Retrieval-Augmented Generation: A Survey, May 2024, arXiv
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Articles
- ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation
- Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
- Original paper: *FaithDial: A Faithful Benchmark for Information-Seeking Dialogue*, TACL, December 2022, MIT Press. DOI: 10.1162/tacl_a_00529
Relation to broader trends
Neural information retrieval has evolved from BM25 through dense retrieval and cross-encoder reranking to generative retrieval and LLM agents. With retrieval-augmented generation (RAG) and agentic search, external knowledge access has become an iterative, plannable process — and evaluation has correspondingly shifted from static metrics like nDCG toward process-oriented measures such as task success, citation accuracy, and faithfulness of generated responses. FaithDial belongs to this evaluation-oriented line of work.