When AI Starts to Know Itself: How LLMs Accidentally Learned to Measure Their Own Confidence

Forum topic · ✨步子哥 · 2025-11-12

Summary

Apple researchers report that base (pretrained-only) large language models exhibit surprisingly good semantic calibration—the ability to gauge how likely their answers are to be correct—despite never being explicitly trained for it. By sampling 50 answers per question and clustering them into semantic classes, the team measured semantic confidence and accuracy across models from Qwen, Gemini, Mistral, and Llama (0.5B–72B) on GSM8K, OpenMathInstruct, TriviaQA, and SimpleQA. Base models in concise or full-sentence modes tracked the ideal diagonal closely, and this calibration was independent of model scale. However, instruction tuning (SFT, RLHF, DPO) made models increasingly overconfident—for example, Mistral-7B's calibration error rose from 0.036 to 0.086 (SFT) and 0.254 (DPO). Chain-of-thought prompting caused underconfidence, since answers only crystallize during reasoning. The authors argue calibration is a byproduct of cross-entropy training reaching local loss optimality, and that practices which shift the optimization objective away from honest prediction—like human preference optimization—destroy it. The findings suggest a fundamental capability-calibration trade-off in AI alignment.

When AI Starts to "Know Itself": How LLMs Accidentally Learned to Measure Their Own Certainty

*Decoding semantic calibration: from a sea of tokens to a starry sky of concepts*

> Editor's note: This is a science story about AI "self-awareness." Apple's research team found that base large language models, which appear to merely parrot text, have incidentally acquired a human-intuition-like ability to assess the trustworthiness of their own answers. Surprisingly, when we try to make AI "smarter" through instruction tuning and chain-of-thought reasoning, this natural calibration ability disappears. What mathematical mechanisms lie behind this?

---

The Confident AI vs. the Lost AI

Ask a knowledgeable expert "What is the capital of France?" and they answer "Paris!" instantly and firmly. Ask who won the 2024 Nobel Prize in Physics, and they hesitate: "I think it was... John Hopfield and Geoffrey Hinton?"

The difference: in the first case they *know* they know; in the second, they *know* they're unsure. This awareness of one's own knowledge boundaries is a hallmark of human intelligence.

Now swap in today's most powerful LLMs. They answer fluently on everything from poetry to code debugging, but a fundamental question troubles us: do they really know what they know—and what they don't?

This is not just philosophy. An overconfident AI used in medical diagnosis, legal advice, or financial decisions can cause disasters. AI researchers measure this self-knowledge via calibration: if a model says "80% confident," it should be right 80% of the time when it says so—just like a weather forecaster whose "70% chance of rain" days actually see rain 70% of the time.

The catch: LLMs are fundamentally token predictors, trained to predict the next word—not to assess the correctness of entire answers. Apple's research team made a surprising discovery: viewed at the semantic level rather than the token level, base LLMs show remarkable self-knowledge, even without explicit training. And this "gift" is destroyed by the very techniques—instruction tuning and chain-of-thought—meant to make AI smarter.

Chapter 1: The Art of Calibration

In classification, calibration is easy: an image classifier outputting [0.8, 0.2] is calibrated if it's right 80 times out of 100 on such predictions. But LLMs generate entire sequences. "Paris," "It is Paris," and "The capital of France is Paris" are semantically identical yet token-level different—traditional token-level calibration fails.

Hence semantic calibration. The team's method:

1. Sample 50 answers per question at temperature 1 2. Map each answer to a semantic class via a semantic collapse function B (extract final numeric answers for math; use a strong LLM like Qwen3-14B to canonicalize and cluster open-ended answers) 3. The frequency of the most frequent class is the semantic confidence 4. Check whether that class matches the correct answer → semantic accuracy

If a model's average confidence is ~70% and its accuracy is ~70%, it is semantically calibrated.

The puzzle: LLMs are only trained to minimize next-token cross-entropy loss, never to evaluate answer correctness. Yet the experimental results are striking.

Chapter 2: The Unexpected Gift

The team tested dozens of models from the Qwen, Gemini, Mistral, and Llama families (0.5B–72B parameters) on four datasets—GSM8K (grade-school math), OpenMathInstruct (competition math), TriviaQA, and SimpleQA—in three modes: concise, full-sentence, and chain-of-thought (CoT).

Reliability diagrams show:

Base models + concise/sentence modes: nearly all points hug the diagonal—excellent calibration, independent of scale (even 0.5B models perform well).
Instruction-tuned models (any mode) and base models with CoT: badly calibrated. Instruction-tuned models are systematically overconfident; base models' CoT answers are underconfident.

This yields the first core claim: semantic calibration is an accidental byproduct of standard pretraining. If a model can easily predict its own semantic answer distribution before generating the first token, calibration emerges.

Chapter 3: The Hidden Cost of Instruction Tuning

Comparing three versions of Mistral-7B-v0.1 in sentence mode:

Base model: calibration error 0.036
SFT model: 0.086
DPO model: 0.254

The DPO model becomes extremely overconfident—losing the subtle sense of "I'm not sure" in favor of assertive answers, even when wrong.

The theory: calibration is tied to loss optimality. Base models minimize cross-entropy, a proper loss that encourages honest uncertainty reporting. RLHF/DPO change the objective from "predict accurately" to "score well with humans"—like a student shifting from genuine understanding to exam-cramming, losing sharp awareness of their own blind spots.

LoRA experiments confirm the mechanism: the more easily a small LoRA adapter can learn to predict the model's own semantic answer distribution (small KL gap), the better calibrated the original model.

Chapter 4: The Chain-of-Thought Paradox

On GSM8K, base models achieve a 0.048 calibration error in concise mode but 0.129 in CoT mode—showing systematic underconfidence.

Why? Calibration requires knowing your answer distribution before generating the first token. CoT's essence is "discovering while thinking": the internal reasoning chain may be fragile and error-prone, and answers only crystallize during generation, violating the theory's precondition. Even when the plurality answer is almost always correct, confidence becomes dispersed across sampled chains. CoT's strength—using more compute for better answers—is precisely what destroys calibration.

Chapter 5: Scale-Independence

Counterintuitively, among base models in concise/sentence mode, there is almost no correlation between capability (semantic accuracy) and calibration error—a 0.5B model can be as well calibrated as a 70B model.

This contrasts with prior next-token calibration studies in multiple-choice settings, where larger base models calibrate better. The team's explanation: all base models share the same training objective (cross-entropy minimization). Once a model reaches local loss optimality—unable to reduce loss further via simple perturbations—calibration follows, regardless of scale. Calibration is not a byproduct of capability but a structural feature of the training objective.

Chapter 6: Why LLMs Differ from Other Deep Networks

CNNs and ViTs in computer vision show inconsistent calibration—some severely overconfident. Yet *all* base LLMs calibrate well (outside CoT settings). The key is subtle training-practice differences:

LLM training: validation loss is closely monitored; training stops as soon as loss rises (overfitting), landing at the loss-minimizing frontier.
Image classifier training: practitioners track error rates and keep training even when validation loss rises, drifting models away from loss optimality.

Since calibration is equivalent to loss optimality (Theorem 7), LLM training practice accidentally preserves calibration, while vision practice accidentally destroys it.

Chapter 7: Limits of the Theory

The authors acknowledge limitations:

1. Calibration type: they study sampling-based semantic calibration (50 samples per question—expensive, not real-time). Whether verbalized confidence ("I'm 80% sure") also emerges remains open. 2. Datasets: only 4 datasets; e.g., TruthfulQA (common human misconceptions) may behave differently since it violates the in-distribution assumption. 3. Formalization gaps: the theoretical chain is (C)→(D) calibration ⇔ local loss optimality (rigorously proven); (B)→(C) base models optimal over easy-to-learn perturbations (partially proven, relying on hypotheses from Błasiok et al.); (A)→(B) WB-perturbations easy to learn iff the model knows its B-confidence (proven in a circuit-complexity version). The overall chain relies on a heuristic notion of "easy to learn." 4. Practicality: multi-sample confidence estimation may be too slow in deployment.

Chapter 8: Does AI Have an "Intrinsic Sense of Probability"?

Base LLMs were never taught to "assess their own certainty"—they only learned to predict the next word. Yet something like metacognition emerged spontaneously. This suggests that sensing uncertainty is inherent to intelligence: modeling language's statistical structure inevitably requires modeling ambiguity. For "capital of France," 99% of corpus answers are "Paris" → high confidence. For hard math, multiple paths and common errors exist → a learned "sense of distribution."

Instruction tuning may destroy this because (1) feedback signals are sparse (good/bad, no confidence labels), (2) the objective shifts from honest prediction to pleasing users, and (3) instruction data shifts the distribution. CoT fails because reasoning chains are fragile, cognitive resources go to reasoning rather than self-assessment, and answers emerge late.

Epilogue: Between Certainty and Uncertainty

So, do LLMs "know what they don't know"? Base models do possess a primitive, intuitive self-knowledge—not designed but an emergent byproduct of the training objective. But our efforts to make AI more useful—instruction tuning for compliance, CoT for reasoning—may weaken this self-awareness, creating a calibration-capability trade-off:

Base models: calibrated but limited
Instruction-tuned models: capable but overconfident
CoT models: strong reasoners but underconfident

Promising future directions: calibration-preserving fine-tuning, efficient confidence estimation without multi-sampling, multi-distribution calibration, and distinguishing intrinsic vs. verbalized confidence.

True intelligence begins with self-knowledge: the brightest stars are not necessarily the largest, but those that can honestly state the limits of their own light.

References

1. Nakkiran, P., et al. (2025). *Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs*. arXiv:2511.04869v1. 2. Kadavath, S., et al. (2022). Language Models (Mostly) Know What They Know. *NeurIPS*. 3. Farquhar, S., et al. (2024). Semantic Entropy Probes Linguistic Uncertainty. *ICLR*. 4. Gopalan, P., et al. (2024). Loss minimization yields calibration for most neural networks. *ICML*. 5. Błasiok, J., et al. (2023b). Calibration, Entropy, and Generalization in Neural Networks. *NeurIPS*. 6. Błasiok, J., & Nakkiran, P. (2024). SmoothECE: Principled Calibration Error Estimation. *ICLR*. 7. Hu, E. J., et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. *ICLR*. 8. Wei, J., et al. (2024). SimpleQA: A Simple Question Answering Benchmark. *ArXiv*. 9. Toshniwal, S., et al. (2025). OpenMathInstruct-2: A Massive Math Instruction Tuning Dataset. *ArXiv*.