Multi-Turn Multi-Modal Question Clarification for Enhanced Conversational Understanding

This post summarizes the arXiv paper Multi-Turn Multi-Modal Question Clarification for Enhanced Conversational Understanding (February 2025) by Kimia Ramezan, Alireza Amiri Bavandpour, Yifei Yuan, Clemencia Siro, and Mohammad Aliannejadi.

Key points

• Task: The paper introduces the Multi-turn Multi-modal Clarifying Questions (MMCQ) task, which combines text and visual modalities to refine user queries progressively over multiple conversational turns.
• Motivation: Traditional conversational query clarification relies on text-only questions, which fail to capture complex user preferences—especially those involving visual attributes. Prior single-turn multi-modal methods do not support progressive intent refinement across turns.
• Dataset: The authors create ClariMM, a large-scale dataset comprising over 13k multi-turn interactions and 33k question-answer pairs containing multi-modal clarifying questions.
• Model: They propose Mario, a retrieval framework with a two-phase ranking strategy: initial retrieval with BM25, followed by a multi-modal generative re-ranking model that integrates textual and visual information from the conversational history.
• Results: Multi-turn multi-modal clarification outperforms uni-modal and single-turn approaches, improving MRR by 12.88%. Gains are most significant in longer interactions, demonstrating the value of progressive refinement for complex queries.

Original abstract

> Conversational query clarification enables users to refine their search queries through interactive dialogue, improving search effectiveness. Traditional approaches rely on text-based clarifying questions, which often fail to capture complex user preferences, particularly those involving visual attributes. While recent work has explored single-turn multi-modal clarification with images alongside text, such methods do not fully support the progressive nature of user intent refinement over multiple turns. Motivated by this, we introduce the Multi-turn Multi-modal Clarifying Questions (MMCQ) task, which combines text and visual modalities to refine user queries in a multi-turn conversation. To facilitate this task, we create a large-scale dataset named ClariMM comprising over 13k multi-turn interactions and 33k question-answer pairs containing multi-modal clarifying questions. We propose Mario, a retrieval framework that employs a two-phase ranking strategy: initial retrieval with BM25, followed by a multi-modal generative re-ranking model that integrates textual and visual information from conversational history. Our experiments show that multi-turn multi-modal clarification outperforms uni-modal and single-turn approaches, improving MRR by 12.88%. The gains are most significant in longer interactions, demonstrating the value of progressive refinement for complex queries.

Context and implications

• Architecture: Cascaded retrieval + re-ranking + generation remains the mainstream pattern; agentic paradigms increasingly treat retrieval frequency and strategy as learnable decisions.
• Data: High-quality instruction data and interaction logs remain critical; synthetic data requires care to avoid knowledge leakage and distribution shift.
• Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation.
• Deployment: Latency, cost, interpretability, and safety are hard constraints for production systems—academic benchmarks alone are insufficient.

Engineering checklist

| Item | Question | Suggestion | |------|----------|------------| | Data | Does training/index contain PII? How is versioning managed? | Partitioned indexes, sanitization, rollback-able embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascade + early stop, cache hot queries, async re-ranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation checks | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Per-query token and GPU cost? | Route to smaller models, distillation, hybrid sparse+dense |

Note: quantitative details should be verified against the original PDF before citing.