Conversational vs Traditional: Comparing Search Behavior and Outcome in Legal Case Retrieval (SIGIR 21 Short Paper)

Overview

This is a short paper published at SIGIR 2021, the premier conference on information retrieval. It investigates a direct comparison between conversational search and traditional (keyword-based) search in the domain of legal case retrieval.

• Title: Conversational vs Traditional: Comparing Search Behavior and Outcome in Legal Case Retrieval
• Venue: SIGIR 2021 (short paper)
• Source: https://dl.acm.org/doi/abs/10.1145/3404835.3463064
• Topic area: Conversational information seeking / vertical search (legal IR)

Why This Matters

Legal case retrieval is a challenging vertical for search systems:

• Complex information needs — legal professionals search for precedents where relevance depends on jurisdiction, facts, and procedural history, not just keyword overlap.
• Multi-turn refinement — real legal research rarely fits in a single query; users iteratively narrow or pivot their needs.
• Interface paradigm questions — conversational systems promise natural multi-turn interaction, but it is unclear whether this translates into better search behavior and outcomes compared to well-honed keyword search interfaces.

This paper addresses that question empirically by comparing user search behavior (how people formulate queries and interact) and search outcomes (retrieval effectiveness / task success) across the two paradigms in a legal case retrieval setting.

Key Points

• The paper provides an empirical comparison of conversational vs. traditional search interfaces, rather than proposing a new retrieval model.
• The domain is legal case retrieval, a vertical where query complexity and user expertise make the conversational-vs-keyword question especially interesting.
• Both behavioral measures (interaction patterns, query reformulation, follow-up behavior) and outcome measures (retrieval effectiveness / task performance) are considered.
• The work contributes evidence for the broader debate in the IR community on when conversational interaction genuinely helps professional search tasks.

Scope and Caveats

• This is a short paper, so the experimental scope and reported details are correspondingly compact.
• Detailed numbers (participant counts, metrics, statistical tests) are available in the original PDF; readers citing quantitative conclusions should consult the source directly via the ACM link above.

Related Reading

• Conversational search and information-seeking dialogues (e.g., work on conversational query reformulation and mixed-initiative interaction)
• Legal IR benchmarks such as COLIEE, which also study statute law and case retrieval
• User studies comparing search interfaces in professional verticals (patent, medical, legal search)

Reader Takeaways

1. Researchers: this is a useful reference for study design when evaluating conversational search against keyword baselines with real users, not just offline benchmarks. 2. Engineers/product teams: interface paradigm choices should be validated against domain-specific user behavior — conversational interaction is not automatically superior. 3. IR community: evaluating both behavior and outcome is essential; effectiveness metrics alone miss how users actually work with the system.