Abstention Is Not One Ruler: The Two Independent Axes of LLM Abstention

You ask an LLM: "Who is the current King of France?"

It confidently answers: "Louis XIX."

Wrong — but not the kind of "wrong answer" wrong. It's the kind of "this question should not have been answered at all" wrong. France has no king. The question is built on a false premise.

Now ask: "What is 2+2?"

It answers: "5."

Also wrong — but this time it's a genuine wrong answer to a valid question.

These two errors both look like "the model gave a wrong response," but they are fundamentally different. One is a wrong answer; the other is a wrong question. Benedikt Wagner (City St George's, University of London), in a July 2026 paper titled *Two Axes of LLM Abstention*, proves that these two errors are orthogonal axes, and existing abstention mechanisms conflate them.

One Ruler Measuring Two Things

The mainstream "selective answering" approach is simple: set a confidence threshold, abstain below it — like a thermometer that raises an alarm at high temperature.

But Wagner found this thermometer measures two different things:

• Answer correctness: Can I answer this question correctly?
• Question answerability: Should this question be answered at all?

The paper defines three classes of questions:

• C (correct-answerable): answerable and answered correctly
• W (wrong-answerable): answerable but answered incorrectly
• U (unanswerable): not answerable (false premises, no solution, etc.)

Ideally the model should abstain on both W and U. But across 5 instruction-tuned models (2B–14B, from 3 model families), Wagner found a disturbing fact:

Model output confidence is nearly blind to question answerability.

Key numbers:

• Output confidence predicting answerability: AUROC 0.54–0.67 (near random)
• Hidden-state linear probes predicting answerability: AUROC 0.97–0.99
• Scaling trend: none. The 14B model is no better at separating answerable/unanswerable questions than the 2B.

Stranger still: even a direction trained to separate C from W (correct vs. incorrect answers) barely separates U from the rest. Answer correctness and question answerability are entangled in the model's representation space — you pull out a direction trying to split them, and they turn out not to lie on the same line at all.

The CREPE Dataset: Real User Questions

Maybe these "unanswerable" questions are artificially constructed and the model just hasn't seen them? Wagner dispelled this using the CREPE dataset.

CREPE questions are real user questions, some with false premises. For example: "Who is the author of the novel *X*, which topped the NYT bestseller list?" — if the novel never topped the list, the question is unanswerable.

On CREPE:

• Trained output readout: near random
• Model self-evaluations (P(IK), P(True), direct premise checking): at best 0.67 AUROC
• Internal linear readouts (hidden-state probes): 0.69–0.77

What does 0.67 mean? A coin flip gives you 0.5. The model's own ability to judge "should this question be answered" is barely better than a coin flip.

A 57% False-Positive Rate: Good Intentions Backfiring

So what if you just tell the model to "check premises" before answering?

Wagner ran that experiment. Result:

The model challenged 57% of perfectly valid questions too.

It can't tell false-premise questions from valid ones, so it questions everything — like an oversensitive smoke alarm that goes off for both cooking and fire, until you just uninstall it.

But Wagner found a fix: route with a linear probe on hidden states. First let the probe judge "is this answerable?" and only trigger premise checking when the probe says unanswerable. The result:

Challenge precision improved 3x.

In other words, the information about answerability *is* in the hidden states (AUROC 0.97–0.99) — the model just won't use it on its own. You have to attach a probe from outside to read it.

Dual-Budget Certification: Making Abstention an Engineering Problem

The paper's most elegant contribution formalizes abstention as "three-class selective acceptance" with two independent budgets:

• alpha_U = 0.15: unanswerable questions incorrectly answered at most 15%
• alpha_W = 0.50: answerable questions answered incorrectly at most 50%
• delta = 0.10: a 90%-confidence statistical guarantee

Certified with exact binomial bounds and validated on a held-out test set independent of training/threshold tuning.

The beauty of this framework: it acknowledges the two error types have different costs and should be controlled separately — like factory QC, where you wouldn't lump "product malfunction" and "wrong shipment" into a single defect rate, because their causes and fixes are entirely different.

An Asymmetry of Scale

The certification results expose a scale-dependent asymmetry:

• Unanswerable budget: 4 of 5 models passed
• Wrong-answer budget: capped by the model's own accuracy

On an 8B model, the factored dual-threshold strategy passed both budgets. On smaller models, the wrong-answer budget hits the model's capability ceiling — no abstention strategy can compensate for a model that simply isn't strong enough.

Practical takeaway: for small models, the abstention ceiling is capability; for large models, it is detection. Beyond some scale, the bottleneck shifts from "can it answer correctly" to "does it know whether the question should be answered."

The Deeper Contribution

This paper does something rare: it shows that a widely used framework (single-threshold confidence abstention) is conceptually wrong — not something tuning can fix.

It's not "the threshold isn't good enough"; it's that "using one ruler to measure two orthogonal things" is the flawed idea itself.

This echoes the Co-Failure Ceiling paper's logic — it's not that "the compositional strategy isn't good enough," it's that "using rho to measure diversity" is the wrong metric. Both papers make the same point: if you pick the wrong dimensions at the conceptual level, no amount of engineering optimization will save you.

Wagner's factored framework turns "abstention" from a vague "when not to speak" question into two clear engineering problems: how to control the mis-answer rate on unanswerable questions, and how to control the error rate on answerable ones. The former relies on detection (hidden-state probes); the latter on capability (the model itself). Both can be optimized and certified independently.

That's far less exciting than "tune one confidence threshold," but it matches the model's actual failure structure. Good engineering is often like this — get the concepts right first, then optimize.

---

Paper: Two Axes of LLM Abstention: Answer Correctness and Question Answerability Author: Benedikt J. Wagner (City St George's, University of London) Date: July 9, 2026