A Simple Question
Ask an LLM: "In which country is the Eiffel Tower?" It answers "France."
Now ask two sub-questions: "In which city is the Eiffel Tower?" → "Paris." "In which country is Paris?" → "France."
The composed sub-answers match the direct answer. That's normal.
But sometimes the LLM answers "France" directly, while the sub-questions yield "Paris" and "Germany." The composed answer — Germany — contradicts the direct one. What does this mean? The model's reasoning is internally inconsistent — it isn't really "reasoning"; it's improvising.
This simple check is the core idea of Operadic Consistency (OC), and its mathematical foundation is operad theory from algebraic topology.
From Intuition to Math: What Is an Operad?
An operad is a formal tool for describing "composable systems." Its central question: how do operations on multiple inputs compose into larger operations?
A cooking analogy:
- "Scramble eggs" is an operation with inputs {eggs, oil, salt}
- "Make a sandwich" is another operation with inputs {bread, lettuce, scrambled eggs}
- "Make a sandwich" can embed "scramble eggs" — the output of one becomes an input to the other
- A "question template" is an operation whose inputs are the blanks to fill
- "Decomposition" splits a complex question into sub-questions, then combines the sub-answers
- Operadic consistency checks whether the direct answer matches the answer reconstructed from decomposition
Operad theory precisely describes this "operation nesting operation" structure.
In LLM reasoning:
Why Is OC Stronger Than Existing Methods?
Existing failure-detection approaches fall into three classes:
1. Self-Consistency: sample multiple times, check answer agreement. Problem: a model may confidently give the same wrong answer every time. 2. Semantic Entropy: measure semantic diversity of answers. Problem: it looks at output diversity, not reasoning structure. 3. P(True): ask the model "Do you think you're right?". Problem: models can be confidently wrong.
OC's uniqueness: it examines the internal structure of reasoning, not surface output statistics. A model can look superficially consistent (same wrong answer every time), but if its sub-reasoning can't compose, OC catches the contradiction.
Detective analogy: self-consistency is "asking the suspect the same question three times"; semantic entropy is "counting how many versions of the story the suspect gives"; P(True) is "asking the suspect 'are you telling the truth?'"; OC is "taking the testimony apart and checking whether the details corroborate each other."
Experimental Results: A Strong Signal Across Models and Datasets
The paper tests 17 models (12 instruction-tuned LLMs + 5 frontier reasoning models) on 5 compositional reasoning benchmarks:
OC–accuracy correlation: on 4 multi-hop QA datasets, the OC rate correlates with accuracy at Pearson r ∈ [0.86, 0.94] (all p ≤ 0.0004). It is the only signal with r ≥ 0.85 on all four datasets. By contrast, chain-of-thought self-consistency (CoT-SC) performs well on HotpotQA and DROP (r = 0.93, 0.87) but collapses to r ≈ 0.45 on MuSiQue and StrategyQA.
Per-question complementarity: after controlling for CoT-SC and semantic entropy, OC still adds significant information (cluster-robust p ≤ 10⁻¹⁶). The result holds even against purpose-built "decomposition-aware" baselines (p ≤ 10⁻¹³).
Selective prediction: at equal compute budget (K=3), OC improves AUARC by +0.086 to +0.096 and AUROC by +0.092 to +0.164 over tuned CoT-SC baselines, with 95% confidence intervals excluding zero.
Reasoning models: on 5 frontier reasoning models, decompositions extracted from the models' own chains of thought yield positive gains in all 16 (dataset, budget, metric) cells under equal-cost comparison, with 12 of 16 confidence intervals excluding zero.
Cross-model accuracy prediction: OC rates can predict accuracy on unseen models — leave-one-out linear regression over the other 11 models achieves a mean absolute error of only 3.0 percentage points, versus 4.6 points (with a bimodal pattern) for CoT-SC at K=10 inference calls.
Why It Matters
OC reveals a deeper insight: LLM reasoning failures are often not random but structural.
When a model contradicts itself between sub-question reasoning and direct answering, this isn't an occasional slip — its reasoning process simply isn't "composing." Direct and decomposed answering follow two paths that never communicate.
This echoes dual-process theory in human cognition: System 1 (intuition) and System 2 (analysis) can give different answers. OC detects exactly the AI version of "System 1 and System 2 disagreeing."
Crucially, OC requires no labels — no ground-truth answers needed, only internal self-consistency of the model's own responses. This makes it nearly free in deployment: 3 inference calls (1 direct + 2 sub-questions) yield a stronger signal than 10-sample self-consistency.
An Honest Assessment
OC has so far been validated only on compositional reasoning tasks (multi-hop QA, math word problems); its applicability to open-ended generation is unclear, and not every question has a natural decomposition.
OC also has false positives and false negatives: a model can be OC-consistent yet wrong (confidently wrong), or OC-inconsistent yet right (following different valid reasoning paths). The paper candidly shows such cases.
The operadic framework currently uses only depth-2 decompositions (one layer of sub-questions); deeper tree-structured decompositions remain to be explored. But as a first step, OC already makes a compelling case: checking the internal structure of reasoning is more effective than checking its surface features.
---
Paper: Operadic consistency: a label-free signal for compositional reasoning failures in LLMs Code: https://github.com/natebottman/operadic-consistency-paper Authors: Nathaniel Bottman (Incubilate), Yinhong Liu (Cambridge), Kyle Richardson (Allen AI) Area: LLM Reasoning, Uncertainty Quantification, Mathematical Foundations