CLUE: Using Large Language Models for Judging Document Usefulness in Web Search Evaluation (CIKM 2025)
- Paper: CLUE: Using Large Language Models for Judging Document Usefulness in Web Search Evaluation
- Venue: CIKM 2025
- Link: https://dl.acm.org/doi/abs/10.1145/3746252.3761158
- Category: Evaluation of Search Engines
- The paper tackles document usefulness judging, one of the most labor-intensive steps in web search evaluation, and investigates whether large language models can perform (or assist with) this task at scale.
- It belongs to the growing LLM-as-judge line of work, where model-based assessors are cross-validated against human judgments rather than trusted blindly.
- IR: Information Retrieval
- RAG: Retrieval-Augmented Generation
- LTR: Learning to Rank
- nDCG: Normalized Discounted Cumulative Gain, a ranking quality metric
- Agentic Search: modeling search as sequential decision-making and tool calling
- ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
- Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv:2401.13178)
- AI Search Has A Citation Problem (CJR, March 2025)
Key points
Context: search evaluation in the LLM era
The forum entry situates CLUE within a broader shift in information retrieval. Classic search stacks form a funnel — recall (candidate generation), precision (reranking), and presentation — but LLM-era systems add two new variables: inference budget and action space (whether to retrieve, how many times, and which tools to call). Evaluation is consequently moving from static metrics like nDCG toward process-oriented measures such as task success rate, citation accuracy, and multi-hop reasoning chain completeness.
Key observations from the entry:
1. Architecture: cascaded retrieve + rerank + generate remains mainstream, but agentic paradigms increasingly make retrieval strategy itself learnable. 2. Data: high-quality instruction data and click/session logs matter as much as model design; synthetic data risks knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM judges need human cross-validation — precisely the problem space CLUE addresses. 4. Product: latency, cost, explainability, and safety are hard constraints that academic benchmarks alone do not capture.
Engineering checklist (from the post)
| Item | Question | Suggestion | |------|----------|------------| | Data | PII in training/index? Versioning? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval steps? | Cascades + early stopping, hot-query caching, async reranking | | Quality | Does offline gain transfer to online CTR/satisfaction? | Interleaving experiments, human audit samples, citation checks | | Safety | Poisoning/bias via open retrieval? | Source allowlists, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Small-model routing, distillation, hybrid sparse+dense |