Agreement Is Not Validity: When Three LLMs Consensus Still Gets It Wrong

A Real-World Scenario

In a K-12 math tutoring session, a student makes a mistake on a fraction addition/subtraction problem. You ask three different LLMs (GPT, Claude, Gemini) to analyze the dialogue and diagnose why the student failed. All three give the same diagnosis: "conceptual understanding gap."

You breathe a sigh of relief — all three models agree, so it must be right, right?

The October 2026 paper "Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue" tells you: don't celebrate yet. Three models agreeing may simply mean they share the same bias.

The Core Problem

Learning Analytics increasingly relies on LLMs to extract information from tutoring dialogues — knowledge tracing, behavior modeling, error diagnosis. But are the models' interpretations actually valid?

A popular validation strategy is "multi-model consensus": if models from different families produce the same label, that label is considered more trustworthy. The underlying assumption is that different models have different biases, so agreement across models with different biases indicates truth.

This paper directly challenges that assumption. The question is not "do models agree with each other?" but "does agreement mean correctness?"

Experimental Design

The research team used three models from different families (one per family), asking them to annotate the same set of K-12 math tutoring dialogues for five failure modes:

1. Uncertainty: the student shows hesitation 2. Misattribution: the student attributes errors to the wrong cause 3. Operator Selection: the student picks the wrong operation (should add but subtracts) 4. Conceptual Gap: missing key conceptual understanding 5. Procedural Slip: knows the concept but executes incorrectly

Annotations were compared against a human-adjudicated reference set. The key question was not each model's individual accuracy, but whether accuracy improves when multiple models agree.

Finding: Consensus Is Not Validity

The paper's core finding in one sentence: inter-model agreement rates are high, but the agreed-upon labels show no significant improvement in alignment with human annotations.

In other words, labels that all three models agree on are no closer to human judgment than a single model's labels.

Why? Because models from different families share similar biases. They have common blind spots in pretraining data, alignment methods, and evaluation paradigms. When they all agree on a label, it may not be because the label is correct — it may be because they all tend to make the same kind of error.

This echoes the concept of a "rationalized shell" — models are not "diagnosing" but "generating plausible-looking diagnoses." Three models producing the same rationalized shell doesn't mean the truth lies beneath it.

Asymmetric Performance Across Failure Modes

The paper also found an asymmetry: models are more "confident" but less accurate on certain failure modes.

For vague, high-level labels like "conceptual gap", models tend to over-attribute — labeling many errors as conceptual gaps. This diverges from human experts, who tend to first rule out more specific errors like "procedural slip."

For concrete, verifiable labels like "operator selection", models perform better. This suggests models are more reliable on tasks requiring concrete reasoning and more prone to "rationalized shells" on tasks requiring ambiguous judgment.

What This Means

For LLM-as-judge: Multi-model consensus is an overrated validation strategy. Consensus can rule out "one model's unique bias," but cannot rule out "the bias shared by all models." The latter is the more dangerous one.

For learning analytics: When using LLMs to annotate educational dialogues, you cannot rely on inter-model agreement alone. A human-adjudicated reference set as ground truth is mandatory — otherwise you may be optimizing a wrong metric.

For AI evaluation methodology: The paper offers a broader lesson — agreement and accuracy are two different dimensions. Reporting only agreement rates severely overestimates annotation quality, just as reporting only recall overestimates audit quality.

Cross-Domain Analogy: Three Witnesses All Saw a Red Car

Imagine a car accident where three witnesses all say they saw a red car run a red light. The judge breathes a sigh of relief — all three witnesses agree.

But the investigation reveals: the accident happened at dusk, when red and brown are hard to distinguish. All three witnesses saw a brown car and perceived it as red. They agreed — and they were all wrong.

"Three models all agree" is the same as "three witnesses all saw a red car" — consensus can come from shared genuine perception, or from shared perceptual deficits. Without ground truth, you cannot tell the two apart.

A Deeper Question

This paper touches the fundamental dilemma of AI evaluation: what do we use to validate AI output?

If we validate against human annotations — human annotations themselves are noisy. If we validate with another AI — we're back to "consensus ≠ validity." If we validate with experimental outcomes — many tasks have no executable ground truth.

Perhaps we need to accept one fact: LLM outputs are hypotheses, not conclusions. A model's label is a hypothesis awaiting verification, not an answer that can be used directly. Multi-model consensus can raise a hypothesis's priority, but it cannot turn a hypothesis into a conclusion.

This belongs to the same lineage as the "proxy goal trap" — we optimize "inter-model agreement rate," but agreement rate is not the "validity" we actually want. There is a gap between the two concepts, and we often pretend not to see it.

---

Paper: Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue Code: claytoncohn/LAK27_Supplementary_Materials Authors: Clayton Cohn, Joyce Fonteles, Kirk Vanacore, Gianni Mazza, Candida Crawford