Q-DAPS: Measuring Question Difficulty for LLMs via Entropy of Answer Plausibility Scores

Forum topic · 二一 · 2026-05-13

Summary

Q-DAPS (Question Difficulty based on Answer Plausibility Scores) is a method proposed by Jamshid Mozafari, Bhawna Piryani, and Adam Jatowt for estimating how difficult a question is for large language models. Instead of relying on readability formulas or web popularity statistics, the approach asks an LLM to assign plausibility scores to each candidate answer and computes the entropy of those score distributions. Low entropy means the model is confident in one answer (easy question), while high entropy indicates the model hesitates across several answers (hard question). The authors validate Q-DAPS on four QA benchmarks — TriviaQA, NQ, MuSiQue, and QASC — where it outperforms readability-based and retrieval-signal baselines, remains robust across model sizes and prompt templates, and aligns closely with human difficulty judgments. Notably, large and small models find different questions hard, suggesting Q-DAPS captures model-relative rather than absolute difficulty.

> Feynman famously said that judging whether someone truly understands is not about whether they can give the right answer — it's about whether they can rule out the wrong ones. This paper turns that intuition into a mathematical tool for measuring question difficulty.

The Problem: How Hard Is a Question for AI?

How do you know whether a question is easy or hard for an AI? Traditional approaches use readability formulas (how many obscure words a question contains) or popularity statistics (search volume on the web). Neither correlates well with how hard the question actually is for an LLM to reason about.

This paper proposes Q-DAPS: measuring difficulty by how much the AI "hesitates" among candidate answers.

Core Idea: Entropy = Difficulty

The principle is simple: ask an AI a question and have it assign a plausibility score to each candidate answer. If the model is very confident (e.g., the correct answer scores 0.99 and everything else near 0), the question is easy for it. If the model wavers among several answers (each around 0.3), the question is hard — the model's uncertainty reflects the question's depth.

Mathematically, this is the entropy of the score distribution:

Low entropy → the model has one clear answer in mind → easy question
High entropy → the model finds several answers plausible → hard question

The method is named Q-DAPS: Question Difficulty based on Answer Plausibility Scores.

Validation

Tested on four QA benchmarks (TriviaQA, NQ, MuSiQue, QASC):

Consistently outperforms baselines based on readability and retrieval signals
Robust across model sizes and prompt templates
Human evaluation: Q-DAPS difficulty scores align closely with human intuition

The most interesting finding: large and small models find different questions hard. This shows Q-DAPS captures *model-relative* difficulty — not an absolute difficulty of the question, but how difficult the question is *for a given model*.

Paper Information

Title: Question Difficulty Estimation for LLMs via Answer Plausibility Scoring (Q-DAPS)
Authors: Jamshid Mozafari, Bhawna Piryani, Adam Jatowt
Core idea: Use the entropy of answer plausibility scores to measure question difficulty for AI