Evaluating Search Relevance Part 2: Using Phi-3 as a Relevance Judge (Elastic)

Forum topic · 小凯 · 2026-07-05

Summary

This post discusses part 2 of Elastic's search relevance evaluation series, which explores practical experience using Microsoft's Phi-3 small language model family as an automated relevance judge for search results. The approach addresses a core challenge in information retrieval: manually assessing whether retrieved documents actually satisfy a query is slow and expensive, while relying solely on offline metrics like nDCG may not reflect real user satisfaction. Using an LLM as a judge offers a scalable middle ground, generating relevance judgments for query-document pairs that can be used to build or augment evaluation datasets for ranking pipelines (BM25, dense retrieval, rerankers). The forum post frames this within the broader LLM-era search stack—retrieval, reranking, generation, and evaluation—emphasizing that LLM-as-judge outputs should be cross-validated against human assessments, and that latency, cost, and trustworthiness of judgments are key engineering constraints. Note: much of the forum post is template boilerplate; readers should consult the original Elastic Search Labs article for concrete methodology and quantitative results.

Evaluating Search Relevance Part 2: Using Phi-3 as a Relevance Judge (Elastic)

This forum post indexes a blog article from Elastic's Search Labs series on search relevance evaluation. The original article is available at:

https://www.elastic.co/search-labs/blog/evaluating-search-relevance-part-2

Key points

Topic: Part 2 of Elastic's series on evaluating search relevance, focusing on using the Phi-3 LLM family (Microsoft's small language models) as an automated relevance judge for search result evaluation.
Motivation: Manual relevance judgments are expensive and slow to produce at scale, while purely automatic metrics (e.g., nDCG against sparse labels) may not capture actual user-perceived relevance. An LLM judge offers a scalable way to generate relevance labels for query–document pairs.
Use case: Feeding queries and retrieved documents to Phi-3 to obtain graded relevance assessments, which can then populate or extend evaluation datasets used to benchmark ranking pipelines (BM25, dense retrieval, rerankers).
Practical caveats (per the forum poster's framing): LLM-as-judge scores should be cross-validated with human judgments; judgment quality, inference cost, and latency are the main engineering trade-offs when deploying a small model like Phi-3 for evaluation workloads.

Context within the search stack

The forum post situates the article in the modern retrieval pipeline: query understanding → recall (sparse/dense hybrid) → reranking → generation (RAG) → evaluation. LLM-as-judge belongs to the evaluation layer, complementing:

Offline IR metrics (nDCG@10, MRR, Recall@k)
Human-labeled test collections
Online interleaving / A/B experiments

Note on this forum post

Most sections of the forum post (methodology, experiments, ablations) are generic template text rather than a faithful summary of the Elastic article. For the actual prompt design, model setup, evaluation protocol, and quantitative comparisons against human judgments, consult the original Elastic Search Labs post linked above.

Evaluating Search Relevance Part 2: Using Phi-3 as a Relevance Judge (Elastic)

Summary

Evaluating Search Relevance Part 2: Using Phi-3 as a Relevance Judge (Elastic)

Key points

Context within the search stack

Note on this forum post

Further reading (cross-referenced in the forum)

Tags