Evaluating Search Relevance Part 2: Using Phi-3 as an LLM Relevance Judge - Elasticsearch Search Labs

Forum topic · 小凯 · 2026-07-05

Summary

This article is part 2 of Elastic's Search Labs series on evaluating search relevance, focusing on using Microsoft's Phi-3 small language model family as an automated relevance judge. Instead of relying solely on expensive and slow human judgment or large proprietary LLMs, the Elastic team explores practical experience running Phi-3 to grade how well search results match user queries. The post covers the motivation for LLM-as-a-judge workflows in search quality evaluation, prompt design for relevance grading, and lessons learned from applying a compact open-weight model to real retrieval evaluation tasks. Key themes include the trade-offs between model size, cost, latency, and judgment quality; how small LLMs can scale relevance annotation across large query-document datasets; and how automated judgments fit into broader search relevance evaluation pipelines alongside metrics like nDCG and human-labeled benchmarks. The article is a practitioner-oriented reference for engineering teams building LLM-based evaluation for retrieval systems, and serves as a bridge between classic information retrieval metrics and modern generative-model-assisted assessment.

This post is part 2 of Elastic's Search Labs article series on evaluating search relevance, and covers practical experience using the Phi-3 LLM family as an automated relevance judge.

> Source: Evaluating search relevance part 2 - Phi-3 as relevance judge

Background

Evaluating whether search results actually match user intent is a persistent challenge in information retrieval. Traditional approaches rely on:

Human judgments — accurate but slow and expensive to scale
Offline metrics such as nDCG@10 and MRR computed over labeled datasets
Large proprietary LLMs as judges — effective but costly

This article investigates a middle path: using Phi-3, a compact open-weight model family from Microsoft, to automatically grade query–document relevance.

Key points

The post documents hands-on experience using Phi-3 as an LLM-as-a-judge for search relevance evaluation within Elasticsearch workflows.
Small language models offer attractive trade-offs on cost, latency, and self-hosting, making large-scale automated relevance annotation more practical than relying solely on frontier commercial models.
Prompt design for relevance grading is a central concern: the model must be instructed to judge whether a document answers the query, ideally producing structured judgments usable in ranking metrics.
Automated LLM judgments are positioned as a complement to — not a replacement for — human evaluation; cross-validation against human labels remains important for trustworthiness.
The series context (part 2) implies a broader methodology for search quality evaluation, where automated judging feeds into relevance metrics and regression testing for search deployments.

Why it matters for search engineers

1. Scalability: LLM judges can annotate orders of magnitude more query–document pairs than human raters, enabling broader evaluation coverage. 2. Cost control: Running a small open-weight model like Phi-3 locally or on modest hardware dramatically reduces per-judgment cost compared to API-based frontier models. 3. CI/CD for search: Automated relevance judgments enable regression testing whenever retrieval logic, analyzers, or ranking models change. 4. Caveats: LLM judges can exhibit biases (position bias, verbosity bias) and may disagree with humans on ambiguous or domain-specific queries; audit samples regularly.

Takeaways

Compact open-weight LLMs are now viable relevance judges for practical search evaluation pipelines.
Treat LLM-as-judge output as one signal in a broader evaluation stack that still includes human judgments and classic IR metrics.
The approach fits naturally into Elasticsearch-centric relevance workflows, supporting iterative improvement of retrieval quality.

*Note: This page summarizes the Elastic Search Labs blog post based on available metadata; consult the original article for exact prompts, model configurations, and quantitative results.*