Enhancing Discoverability in Enterprise Conversational Systems with Proactive Question Suggestions

This post introduces an academic paper indexed under the Search Assistance section: arXiv:2412.10933 (December 14, 2024).

Authors / Affiliation: Xiaobin Shen, Daniel Lee, Sumit Ranjan, Sai Sree Harsha, Pawan Sevak, Yunyao Li.

Key points

• Problem: Enterprise conversational AI systems are increasingly popular for assisting users with daily tasks such as marketing and customer management. New users often struggle to ask effective questions, especially in emerging systems with unfamiliar or evolving capabilities.
• Approach: The paper proposes a framework that generates proactive, context-aware question suggestions, aiming to address immediate user needs while improving feature discoverability.
• Method: The framework combines periodic user intent analysis at the population level with chat-session-based question generation.
• Evaluation: The framework was evaluated using real-world data from the AI Assistant for Adobe Experience Platform (AEP), demonstrating improved usefulness and system discoverability.

Original abstract

> Enterprise conversational AI systems are becoming increasingly popular to assist users in completing daily tasks such as those in marketing and customer management. However, new users often struggle to ask effective questions, especially in emerging systems with unfamiliar or evolving capabilities. This paper proposes a framework to enhance question suggestions in conversational enterprise AI systems by generating proactive, context-aware questions that try to address immediate user needs while improving feature discoverability. Our approach combines periodic user intent analysis at the population level with chat session-based question generation. We evaluate the framework using real-world data from the AI Assistant for Adobe Experience Platform (AEP), demonstrating the improved usefulness and system discoverability of the AI Assistant.

Context within search / recommendation / personalization

The post situates this work at the intersection of conversational systems and large-scale search/recommendation, highlighting broader trends:

• Architecture: Cascaded retrieval + rerank + generation remains mainstream, while agentic paradigms increasingly treat retrieval frequency and strategy themselves as learnable decisions.
• Data: High-quality instruction data and click/session logs are both critical; synthetic data must guard against knowledge leakage and distribution shift.
• Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation.
• Production constraints: Latency, cost, explainability, and safety policies are hard constraints for industrial deployment.

Related entries (cross-references from the list)

• Asking Clarification Questions to Handle Ambiguity in Open-Domain QA (arXiv:2305.13808)
• Asking Clarifying Questions in Open-Domain Information-Seeking Conversations (DOI: 10.1145/3331184.3331265)
• DiAL: Diversity aware listwise ranking for query auto-complete (EMNLP)
• Evaluating auto-complete ranking for diversity and relevance (ECIR)
• Evaluation and Continual Improvement for an Enterprise AI Assistant (arXiv:2407.12003)
• Generating Query Recommendations via LLMs (arXiv:2405.19749)

Suggested reading checklist

• Researchers: Replicate core comparisons and check whether statistical significance and compute costs are reported.
• Engineers: Extract pluggable modules (encoders, rerankers, planners) and evaluate integration cost with existing stacks.
• Product managers: Focus on user-perceivable benefits (latency, answer trustworthiness, multi-turn consistency) rather than offline nDCG alone.

*Note: Specific quantitative results should be verified against the original PDF; this post is based on the abstract and public metadata.*