Q-DAPS: Measuring Question Difficulty for LLMs via Entropy of Answer Plausibility Scores

Forum topic · 二一 · 2026-05-13

Summary

A Chinese forum post discusses Q-DAPS (Question Difficulty based on Answer Plausibility Scores), a method by Jamshid Mozafari, Bhawna Piryani, and Adam Jatowt for estimating how difficult a question is for large language models. Instead of relying on readability formulas or search popularity, Q-DAPS asks an LLM to assign plausibility scores to candidate answers and computes the entropy of those scores. Low entropy means the model is confident in one answer (easy question); high entropy means the model is split among candidates (hard question). Evaluated on four QA benchmarks—TriviaQA, NQ, MuSiQue, and QASC—the approach outperforms readability- and retrieval-signal-based baselines, is robust across model sizes and prompt templates, and aligns well with human judgments of difficulty. A key finding is that large and small models find different questions hard, showing Q-DAPS captures model-relative rather than absolute difficulty. The paper is titled 'Question Difficulty Estimation for LLMs via Answer Plausibility Scoring (Q-DAPS)'.

A forum post on zhichai.net introduces a paper on estimating question difficulty for large language models, framed with a quote attributed to Feynman: knowing whether someone truly understands is less about giving the right answer and more about ruling out wrong ones.

The Problem

How do you know whether a question is easy or hard for an AI? Traditional approaches use readability formulas (how obscure the wording is) or popularity statistics (search volume), but these correlate poorly with actual reasoning difficulty for LLMs.

Core Idea: Entropy = Difficulty

Q-DAPS (Question Difficulty based on Answer Plausibility Scores) works as follows:

1. Ask an LLM a question and have it assign plausibility scores to each candidate answer. 2. Compute the entropy of these scores:

Low entropy → the model concentrates probability on one answer → the question is easy.
High entropy → the model hedges across several answers → the question is hard.

The intuition: the model's uncertainty reflects the question's depth. If the correct answer scores 0.99 and everything else near 0, the question is trivial for that model; scores spread around 0.3 indicate a genuinely difficult question.

Validation

Tested on four QA benchmarks — TriviaQA, NQ, MuSiQue, and QASC:

Consistently outperforms baselines based on readability and retrieval signals
Robust across model sizes and prompt templates
Human evaluator studies confirm Q-DAPS difficulty scores align well with human intuition

Notably, large and small models find different questions hard, meaning Q-DAPS captures model-relative difficulty rather than an absolute notion of a question's hardness.

Paper Information

Title: Question Difficulty Estimation for LLMs via Answer Plausibility Scoring (Q-DAPS)
Authors: Jamshid Mozafari, Bhawna Piryani, Adam Jatowt
Core idea: Use the entropy of answer plausibility scores to measure a question's difficulty for AI models.