No Single Best Model for Diversity: Learning a Router to Select LLMs for Diverse Open-Ended Responses

Forum topic · 小凯 · 2026-04-04

Summary

A 2025 arXiv paper (2504.01256) by Yuhan Liu, Fangyuan Xu, and Vishakh Padmakumar studies how large language models can comprehensively generate valid answers to open-ended prompts that permit many acceptable responses. The authors introduce diversity coverage, a metric measuring the total quality scores assigned to each unique answer in a predicted answer set relative to the best possible answer set of the same size. Evaluating 18 LLMs with this metric, they find no single model dominates at producing diverse responses across a wide range of prompts, yet for each individual prompt there exists a model that significantly outperforms all others at generating diverse answer sets. Motivated by this, they train a router that predicts the best model per query. On NB-Wildchat, the trained router beats the single-best-model baseline (26.3% vs 23.8% diversity coverage), and it generalizes to an out-of-domain dataset (NB-Curated) as well as to different answer-generation prompting strategies. The work lays a foundation for research on generating comprehensive answer sets when a pool of models is available.

Paper Overview

Field: NLP
Authors: Yuhan Liu, Fangyuan Xu, Vishakh Padmakumar
Published: 2025-04-01
arXiv: 2504.01256

Key Points

When prompts permit a large number of valid answers, comprehensively generating them is the first step toward satisfying a wide range of users. This paper studies methods to elicit a comprehensive set of valid responses.

The authors introduce diversity coverage, a metric that measures the total quality scores assigned to each unique answer in a predicted answer set, relative to the best possible answer set with the same number of answers.
Using this metric, they evaluate 18 LLMs and find that no single model dominates at generating diverse responses across a wide range of open-ended prompts.
However, per prompt, there exists a model that significantly outperforms all other models at generating a diverse answer set.
Motivated by this finding, they introduce a router that predicts the best model for each query.
On NB-Wildchat, the trained router outperforms the single-best-model baseline (26.3% vs 23.8%).
The router further generalizes to an out-of-domain dataset (NB-Curated) and to different answer-generation prompting strategies.

The work lays a foundation for researching comprehensive answer generation when a set of models is available.

Original Abstract (excerpt)

> When posed with prompts that permit a large number of valid answers, comprehensively generating them is the first step towards satisfying a wide range of users. In this paper, we study methods to elicit a comprehensive set of valid responses. To evaluate this, we introduce diversity coverage, a metric that measures the total quality scores assigned to each unique answer in the predicted answer set relative to the best possible answer set with the same number of answers. Using this metric, we evaluate 18 LLMs, finding no single model dominates at generating diverse responses to a wide range of open-ended prompts. Yet, per each prompt, there exists a model that outperforms all other models significantly at generating a diverse answer set. Motivated by this finding, we introduce a router that...

*Auto-collected on 2026-04-04*