Are Large Language Models Consistent over Value-laden Questions? (arXiv 2407.02996)

Forum topic · 小凯 · 2026-07-05

Summary

This July 2024 arXiv paper by Jared Moore, Tanvi Deshpande, and Diyi Yang (Stanford) investigates whether large language models (LLMs) produce consistent responses when asked value-laden questions, such as those probing moral, political, or social values. The paper (arXiv:2407.02996) examines consistency at two levels: test-retest consistency (whether the same model gives the same answer to an identical question asked repeatedly) and paraphrase consistency (whether the model maintains the same stance when questions are reworded). The motivation is that many studies use survey-style questionnaires to characterize the values, political leanings, or moral judgments of LLMs, but such measurements are only meaningful if the models respond consistently. The work highlights that apparent inconsistencies complicate the interpretation of value measurements in LLMs and raises concerns about relying on single-question probing for evaluating model alignment, fairness, and safety. The forum post frames the paper within the broader context of LLM evaluation, information retrieval, and search/recommendation systems, emphasizing reproducible benchmarks and open questions in evaluation methodology. Readers are advised to consult the original PDF for quantitative results.

Are Large Language Models Consistent over Value-laden Questions? (arXiv 2407.02996, July 2024)

Metadata

| Field | Content | |-------|---------| | Title | Are Large Language Models Consistent over Value-laden Questions? | | Authors | Jared Moore, Tanvi Deshpande, Diyi Yang | | Published | July 2024 (arXiv) | | Link | https://arxiv.org/abs/2407.02996 | | Type | Academic paper |

Overview

Many researchers and practitioners characterize the values, moral stances, and political leanings of large language models by presenting them with survey-style questionnaires. This paper asks a prerequisite question: are LLMs even *consistent* in how they answer value-laden questions? If a model gives different answers to the same value question under trivially different conditions, then downstream claims about the model's "values" become hard to interpret.

Key points

The study distinguishes between two forms of consistency:
Test-retest consistency: whether the model gives the same answer when the identical question is asked repeatedly (relevant given the stochastic decoding of LLMs).
Paraphrase consistency: whether the model maintains the same stance when a question is rephrased in semantically equivalent ways.
The findings suggest substantial inconsistency in how LLMs respond to value-laden questions, which complicates the interpretation of prior and future work that uses questionnaires to measure LLM values.
The paper issues a methodological warning: single-shot probes of model values may not reflect stable positions, so evaluations of alignment, fairness, and safety should account for response variability.

Why this matters for the IR/search community

Evaluation methodology: As LLM-as-judge and questionnaire-based value audits become common, measurement reliability (consistency) is a prerequisite for validity. This paper is a caution against over-interpreting single-query outputs.
Deployment: For conversational search and recommendation systems that touch sensitive topics, inconsistent value-laden answers translate directly into user-facing reliability problems.
Benchmark design: Consistency checks like those proposed here can be added as a diagnostic layer to any LLM evaluation suite.

Limitations and open questions

Quantitative results, model coverage, and specific datasets should be verified against the original PDF, as this forum summary is based on abstract-level metadata.
Open directions include: how to aggregate noisy responses into reliable value estimates, cross-lingual consistency, and whether fine-tuning or RLHF reduces value-response variance.

Related entries

Evaluation of Retrieval-Augmented Generation: A Survey (arXiv 2405.07437)
ARES: An Automated Evaluation Framework for RAG (arXiv 2311.09476)
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv 2401.13178)

References

Jared Moore, Tanvi Deshpande, Diyi Yang. *Are Large Language Models Consistent over Value-laden Questions?* arXiv:2407.02996, July 2024. https://arxiv.org/abs/2407.02996