No Single Best Model for Diversity: Learning a Router for Comprehensive LLM Responses

Forum topic · 小凯 · 2026-04-04

Summary

This paper (arXiv:2504.01256) studies how to elicit comprehensive sets of valid responses from large language models when prompts permit many valid answers. The authors introduce diversity coverage, a metric measuring the total quality score assigned to unique answers in a predicted set relative to the best possible set of the same size. Evaluating 18 LLMs with this metric shows that no single model dominates at generating diverse responses to open-ended prompts, yet for each prompt there exists a model that significantly outperforms all others. Based on this finding, the authors train a router that predicts the best model per query. On NB-Wildchat, the router outperforms the single-best-model baseline (26.3% vs 23.8%), and generalizes to an out-of-domain dataset (NB-Curated) and to different answer-generation prompting strategies. The work lays a foundation for research on generating comprehensive answer sets from a pool of models.

Paper Overview

Field: NLP Authors: Yuhan Liu, Fangyuan Xu, Vishakh Padmakumar Published: 2025-04-01 arXiv: 2504.01256

Abstract (Translated)

When faced with prompts that allow a large number of valid answers, comprehensively generating them is the first step toward satisfying a broad range of users. This paper studies methods for eliciting a comprehensive set of valid responses. To evaluate this, the authors introduce diversity coverage, a metric that measures the total quality scores assigned to each unique answer in a predicted answer set, relative to the best possible answer set with the same number of answers.

Using this metric, the authors evaluate 18 LLMs and find that no single model dominates at generating diverse responses to a wide range of open-ended prompts. However, for each prompt, there exists a model that significantly outperforms all other models at generating a diverse answer set.

Key Results

No dominant model: Across 18 evaluated LLMs, no single model is best at producing diverse responses to open-ended prompts.
Per-prompt best model exists: For every prompt, there is one model that substantially outperforms all others in diversity coverage.
Router approach: Motivated by this, the authors train a router that predicts the best model for each query.
Results: On NB-Wildchat, the trained router outperforms the single-best-model baseline (26.3% vs 23.8%).
Generalization: The router generalizes to an out-of-domain dataset (NB-Curated) and to different answer-generation prompting strategies.

The work lays a foundation for research on generating comprehensive answer sets when a pool of models is available.