Enhancing Discoverability in Enterprise Conversational Systems with Proactive Question Suggestions

Paper Overview

| Field | Detail | |-------|--------| | Title | Enhancing Discoverability in Enterprise Conversational Systems with Proactive Question Suggestions | | Authors | Xiaobin Shen, Daniel Lee, Sumit Ranjan, Sai Sree Harsha, Pawan Sevak, Yunyao Li | | Published | 2024-12-14 | | Source | https://arxiv.org/abs/2412.10933 | | Type | Research paper | | Category | Search Assistance |

Original Abstract

> Enterprise conversational AI systems are becoming increasingly popular to assist users in completing daily tasks such as those in marketing and customer management. However, new users often struggle to ask effective questions, especially in emerging systems with unfamiliar or evolving capabilities. This paper proposes a framework to enhance question suggestions in conversational enterprise AI systems by generating proactive, context-aware questions that try to address immediate user needs while improving feature discoverability. Our approach combines periodic user intent analysis at the population level with chat session-based question generation. We evaluate the framework using real-world data from the AI Assistant for Adobe Experience Platform (AEP), demonstrating the improved usefulness and system discoverability of the AI Assistant.

Key Points

• Problem: New users of enterprise conversational AI systems struggle to formulate effective questions, particularly when system capabilities are unfamiliar or rapidly evolving.
• Approach: A framework that generates proactive, context-aware question suggestions, combining:
1. Population-level user intent analysis — run periodically to identify common needs and underused features worth surfacing. 2. Chat session-based question generation — suggestions tailored to the current conversation context and immediate user needs.
• Evaluation: Tested on real-world data from the AI Assistant for Adobe Experience Platform (AEP), showing improved suggestion usefulness and overall system discoverability.

Context in the Broader Landscape

The work sits at the intersection of conversational AI and large-scale search/recommendation systems. Relevant themes discussed in the source post include:

• Classic retrieval cascades (recall → re-ranking → generation) remain mainstream, but agentic paradigms are making retrieval strategy itself a learnable decision.
• High-quality instruction data and session logs are as critical as model architecture; synthetic data requires care against distribution shift.
• Offline metrics (nDCG, MRR) increasingly diverge from online satisfaction; LLM-as-judge evaluation needs cross-validation with human assessment.
• Production deployment must respect hard constraints on latency, cost, explainability, and safety, not just benchmark scores.

Engineering Checklist

| Concern | Recommendation | |---------|----------------| | Data | PII partitioning, sanitization, versioned embeddings | | Latency | Cascaded retrieval with early stopping, query caching, async re-ranking | | Quality | Interleaving experiments, human audits, citation verification | | Safety | Source allowlists, adversarial detection, output filtering | | Cost | Model routing, distillation, hybrid sparse+dense retrieval |

Related Work

• Asking Clarification Questions to Handle Ambiguity in Open-Domain QA
• Asking Clarifying Questions in Open-Domain Information-Seeking Conversations (SIGIR)
• DiAL: Diversity aware listwise ranking for query auto-complete (EMNLP)
• Evaluating auto-complete ranking for diversity and relevance (ECIR)
• Evaluation and Continual Improvement for an Enterprise AI Assistant
• Generating Query Recommendations via LLMs

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool invocation | | Gen-IR | Generative Information Retrieval |

> Note: Quantitative results should be verified against the original PDF; this post is based on the abstract and public metadata.