CLUE: Using Large Language Models for Judging Document Usefulness in Web Search Evaluation (CIKM 2025)
This post introduces and analyzes the CIKM 2025 paper CLUE: Using Large Language Models for Judging Document Usefulness in Web Search Evaluation.
- Source: https://dl.acm.org/doi/abs/10.1145/3746252.3761158
- Venue: CIKM 2025
- Category: Evaluation of Search Engines
- Motivation: Web search evaluation has long relied on human relevance judgments, which are expensive and slow at scale. In the LLM era, judging document usefulness is a natural fit for LLM-as-judge approaches, especially as evaluation moves from static metrics like nDCG toward task success, citation accuracy, and multi-hop reasoning quality.
- Positioning: The work sits at the intersection of information retrieval and large-scale search systems, addressing how the responsibilities of retrieval, ranking, generation, and tool calling should be redistributed in the LLM era. In classic search stacks, recall covers coverage, precision ranking handles discrimination, and generation handles presentation; LLM-era systems add inference budget and action space (whether to retrieve, how many times, which tools to call) as new variables.
- Typical method pipeline described in the post: 1. Input & representation: encode queries, documents, and user context into dense/sparse representations or structured prompts; 2. Core modules: retrievers, rerankers, planners, memory modules, and tool interfaces, chained or composed per task; 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, reinforcement learning (including process rewards), and synthetic data bootstrapping; 4. Inference strategies: single-pass retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control.
- Evaluation landscape: Common benchmarks include MS MARCO, BEIR, and Natural Questions; metrics include nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost; baselines include BM25, dense retrieval, cross-encoder rerankers, retrieval-free LLMs, and commercial search APIs. Note: the original post does not reproduce specific numbers from the CLUE paper; readers should consult the PDF before citing quantitative results.
- Insights for search/recommendation: 1. Architecture: cascaded retrieve-rank-generate remains mainstream, but agentic paradigms treat retrieval count and strategy itself as learnable; 2. Data: high-quality instruction data and click/session logs matter as much as model design; synthetic data risks knowledge leakage and distribution shift; 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge must be cross-validated against human assessment; 4. Product: latency, cost, interpretability, and safety are hard constraints for industrial deployment.
- Limitations & open problems: compute-budget-limited experiments, benchmark mismatch with real user distributions, unknown cross-lingual generalization from English-centric data, and safety risks of agentic systems on the open web. Future directions include better test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation.
- IR: Information Retrieval
- RAG: Retrieval-Augmented Generation
- LTR: Learning to Rank
- nDCG: Normalized Discounted Cumulative Gain
- Agentic Search: modeling search as sequential decision-making and tool calling
- Gen-IR: Generative Information Retrieval
- Evaluation of Retrieval-Augmented Generation: A Survey (arXiv 2405.07437)
- ARES: An Automated Evaluation Framework for RAG (arXiv 2311.09476)
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts (arXiv 2105.03011)
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv 2401.13178)
- AI Search Has A Citation Problem, CJR, March 2025
Key points
Engineering checklist (from the post)
| Check | Question | Suggestion | |-------|----------|------------| | Data | PII in training/index? Version control? | Partitioned indexes, sanitization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval depth? | Cascades + early stopping, cache hot queries, async reranking | | Quality | Do offline gains translate to CTR/satisfaction? | Interleaving experiments, human audit samples, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Per-query token and GPU usage? | Route to smaller models, distillation, hybrid sparse+dense |