Validity Without Ground Truth: Using Stated-Preference Economics to Evaluate LLMs

Forum topic · 小凯 · 2026-10-09

Summary

Many questions posed to large language models have no verifiable correct answer, such as valuing a policy or weighing competing values. This paper (arXiv 2610.10506) argues that stated-preference economics offers a general framework for LLM evaluation without ground truth. It adapts concepts including content validity, construct validity, criterion validity, reliability, incentive compatibility, and consequentiality to LLM testing. As a demonstration, the authors administered a published water-quality stated-preference valuation survey (Vossler et al. 2023) to six models. Validity checks took the form of economic theory predictions: demand curves should slope downward, and willingness to pay should respond to the scope of the good and to income. The tests clearly separated models: two older models failed even the most basic checks at a household income of USD 75,000, while the two newest models passed all scoreable theoretical validity tests, though they diverged on convergent validity. The authors caution that passing validity tests shows model responses are coherent, not that they are correct.

Paper Overview

Field: NLP Authors: Daniel Robert Kling Alexander, Catherine Louise Kling Released: 2026-10-07 arXiv: 2610.10506

What the Paper Does

Many questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values. Stated-preference economics has faced this problem for decades, judging survey responses without knowing the true value.

The authors argue that the validity framework from stated-preference economics is a general method for evaluating language models. They map each related concept to LLM evaluation:

Content validity
Construct validity
Criterion validity
Reliability
Incentive compatibility
Consequentiality

Demonstration

The approach is demonstrated using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models.

In this economic application, validity checks take the form of economic theory predictions:

Demand curves should slope downward
Willingness to pay should respond to the scope of the good
Willingness to pay should respond to income

Findings

These tests clearly discriminate between models.
Two older models failed even the most basic checks at a household income level of USD 75,000.
The two newest models passed all theoretical validity tests that could be scored, but diverged on convergent validity.
Importantly, passing validity tests indicates that a model's responses are coherent — not that they are correct.

--- *Collected automatically on 2026-10-09.*