[论文] Benchmark Prediction with Sparse and Unpaired Cached Responses

论文概要

研究领域: ML 作者: Hayden Helm, Avijit Ghosh, Ben Johnson, Carey Priebe 发布时间: 2026-11-02 arXiv: 2611.03614

中文摘要

在所有基准任务的所有查询上评估一个生成模型成本高得无法承受。因此模型常在任务的不同子集、或同一任务的不同查询子集上评估，得到的响应集合与相应的（模型，任务）得分矩阵是稀疏且不成对的。本文提出乘积核透视空间（PKPS）：基于模型嵌入响应的低维表示。PKPS 推广了表示黑盒生成模型的现有方法——通过比较「相似而不仅相同」的查询，因此能够利用稀疏且不成对的信息。理论上我们证明：相对于仅使用稠密且成对的评估数据的方法，基于 PKPS 的基准预测是查询高效的。实证上，在两个大型基准套件上，基于 PKPS 的方法在基准预测任务上优于基线。

原文摘要

Evaluating a generative model on all queries from all available benchmark tasks is prohibitively expensive. As such, models are often evaluated on different subsets of tasks and/or different subsets of queries from a given task. The resulting collection of responses and the corresponding (model, task) score matrix are sparse and unpaired. In this paper, we introduce the Product Kernel Perspective Space (PKPS), a collection of low-dimensional representations of models based on their embedded responses. PKPS generalizes existing methods for representing black-box generative models by comparing responses to similar -- rather than just identical -- queries and is therefore able to utilize sparse and unpaired information. Theoretically, we show that benchmark prediction using PKPS is query-effi...

#论文 #arXiv #ML #小凯