Investigating Users' Search Behavior and Outcome with ChatGPT in Learning-oriented Search Tasks (SIGIR 2024)
This forum entry summarizes a SIGIR 2024 paper that examines how users behave and what outcomes they achieve when using ChatGPT for learning-oriented search tasks.
Metadata
| Field | Content | |-------|---------| | Title | Investigating Users' Search Behavior and Outcome with ChatGPT in Learning-oriented Search Tasks | | Venue | SIGIR 2024 | | Source | https://dl.acm.org/doi/abs/10.1145/3673791.3698406 | | Type | Academic paper | | Section | Task Solving |
One-line Summary
The paper studies users' search behavior and outcomes with ChatGPT in learning-oriented search tasks, providing empirical evidence on how conversational LLM search differs from traditional web search for learning purposes.
Background and Motivation
In large-scale search, recommendation, and personalization systems, information retrieval has long faced challenges around efficiency, scalability, and user intent understanding. Traditional pipeline approaches tend to separate retrieval, ranking, and generation, which makes it hard to meet users' LLM-era expectations for natural language interaction, multi-hop reasoning, and up-to-date knowledge. This paper addresses user-facing behavior in this transition, focusing specifically on learning-oriented search tasks where the goal is not just finding documents but acquiring knowledge.
Core Contributions
- A unified perspective on user search behavior when ChatGPT replaces or augments conventional search engines in learning scenarios.
- A clear decomposition of method components (representation learning, retrievers, re-rankers, planners, generators, feedback mechanisms) relevant to conversational search systems.
- Reproducible evaluation framing that lowers the entry cost for follow-up research on human-LLM search interaction.
- Discussion of interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration.
- Identification of open problems: evaluation credibility, latency and cost, hallucination and safety, cross-lingual and multimodal extension.
- Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora, and public recommendation datasets;
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost;
- Baselines: BM25, dense retrieval, cross-encoder re-ranking, retrieval-free LLMs, and commercial search APIs;
- Ablations: contribution of retrieval steps, re-ranking depth, and training data scale.
- Original paper: Investigating Users' Search Behavior and Outcome with ChatGPT in Learning-oriented Search Tasks, SIGIR 2024. https://dl.acm.org/doi/abs/10.1145/3673791.3698406
Method / System Perspective
Work in this area typically follows a four-step pattern: problem formalization, model/system design, training or construction pipeline, and inference pipeline.
1. Input and representation: encoding queries, documents, and user context into dense or sparse representations, or structured prompts; 2. Core modules: retrievers, re-rankers, planners, memory modules, and tool interfaces, chained or run in parallel; 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, reinforcement learning (including process rewards), and bootstrapped data synthesis; 4. Inference strategies: single-pass retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control.
Evaluation
Typical evaluation setups in this research area include:
For this paper's specific quantitative results, please consult the original PDF via the ACM DL link above; this summary is based on the abstract and public metadata.
Key Insights for Search / Recommendation / Personalization
1. Architecture: cascaded retrieval + re-ranking + generation remains mainstream, but agentic paradigms are making retrieval count and policy themselves learnable; 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data must guard against knowledge leakage and distribution shift; 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation; 4. Product: latency, cost, explainability, and safety policies are hard constraints for industrial deployment, not just academic benchmark optimization.
Limitations and Future Work
Potential limitations include experiment scale constrained by compute budgets, benchmarks that diverge from real user distributions, English-centric data with unknown cross-lingual generalization, and safety risks of agentic systems on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal and fairness constraints for recommender systems.
Engineering Checklist
| Check | Question | Recommendation | |-------|----------|----------------| | Data | Does training/index data contain PII? How are versions managed? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascade + early stopping, hot-query caching, async re-ranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Per-query token and GPU usage? | Small-model routing, distillation, hybrid sparse+dense retrieval |
Glossary
| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool invocation | | Gen-IR | Generative Information Retrieval |