[论文] Validity Without Ground Truth: What Stated-Preference Economics Offers...

论文概要

研究领域: NLP 作者: Daniel Robert Kling Alexander, Catherine Louise Kling 发布时间: 2026-10-07 arXiv: 2610.10506

中文摘要

现在向大语言模型提出的许多问题并没有可对标的正确答案：一项政策的价值、用户应该选择哪个选项、如何权衡相互竞争的价值观。陈述偏好经济学已经面对这个问题数十年。它在不知道真实价值的情况下判断问卷回答，通过一个有效性框架及相关概念：内容效度、构念效度、效标效度、信度、激励相容性和后果性。我们认为这个框架是评估语言模型的一种通用方法，并阐述了每个概念对LLM评估的含义。我们使用一份已发表的水质陈述偏好经济评估问卷（Vossler等人，2023）对六个模型进行了演示。在这个经济应用中，效度检验采取经济学理论预测的形式：需求曲线应向下倾斜，支付意愿应对物品范围和收入做出反应。这些检验清晰地区分了模型：两个较老的模型在家庭收入75,000美元水平上连最基本的检验都未通过，两个最新的模型通过了我们能评分的所有理论效度检验，但在收敛效度上存在分歧。通过效度检验表明模型的回答是连贯的，但不代表它们是正确的。

原文摘要

Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values. Stated-preference economics has faced this problem for decades. It judges survey responses without knowing the true value, through a framework of validity and related concepts: content, construct, and criterion validity, reliability, incentive compatibility, and consequentiality. We argue that this framework is a general method for evaluating language models, and we set out what each concept means for LLM evaluation. We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models. In this economic application, the valid...

#论文 #arXiv #NLP #小凯