Category Theory Meets Transformers: The Cognitive Categorical Transformer (CCT) Beats GPT-2 Large with Fewer Parameters
> arXiv: 2605.28864v1 | *The Cognitive Categorical Transformer: Category-Theoretic Inductive Biases for Language Modeling* > Author: Al Kari | cs.AI, cs.CL | 2026-05-22
Overview
This forum post discusses a bold experiment: grafting category theory—the mathematics mathematicians jokingly call "abstract nonsense"—directly onto a Transformer. Since "Attention is All You Need" (2017), deep learning has been dominated by a scale-first philosophy. This paper argues instead for structure over scale: the result is the Cognitive Categorical Transformer (CCT), a GPT-2 Small (124M) backbone augmented with ~182M parameters of category-theoretic "cognitive organs."
Headline result:
- CCT (306M params): 21.27 validation PPL on WikiText-103
- GPT-2 Small, matched fine-tune (124M): 24.19
- GPT-2 Large, zero-shot (774M): 22.05
- GT-Full alone: 2.45 PPL improvement (84% of the architectural gain)
- All other components combined: 0.47 PPL (16%)
- Structural priors (add new representational capacity): GT-Full simplicial message passing, PrecisionWeightedPP → ✅ work
- Consistency priors (force agreement with an external rule): sheaf smoothing, adjunction round-trips, curvature regularization → ❌ all fail
- Vaswani, A., et al. (2017). Attention is All You Need. NeurIPS.
- Radford, A., et al. (2019). Language Models are Unsupervised Multitask Learners. OpenAI.
- Al Kari. (2026). The Cognitive Categorical Transformer: Category-Theoretic Inductive Biases for Language Modeling. arXiv:2605.28864v1.
- Bosca, D., & Ghrist, R. (2026). Sheaf discrepancy in ReLU networks.
- Fong, B., & Spivak, D. I. (2019). An Invitation to Applied Category Theory. Cambridge University Press.
- Rao, R. P., & Ballard, D. H. (1999). Predictive coding in the visual cortex. Nature Neuroscience.
CCT outperforms GPT-2 Large using under 40% of its parameters.
What is category theory, briefly?
A category consists of objects, morphisms (structure-preserving maps between objects), and a composition rule with identity morphisms. The intuition: like recipes that differ in ingredients but share the same structure (prepare → transform → combine), category theory captures the shared *shape* of different domains. If words are objects and semantic/grammatical relations are morphisms, language itself may be a category-like structure—which makes category theory a candidate source of inductive biases for language models.
CCT architecture: 182M parameters of "cognitive organs"
| Module | Params | Function | |---|---|---| | GT-Full | ~82M | Simplicial message passing | | Hierarchical Memory | ~48M | Three-level hierarchical memory | | Precision-Weighted PP | ~14.2M | Precision-weighted predictive processing | | Yoneda Self-Model | ~3M | Self-model monitoring | | CausalAttention & aux | ~34.8M | Causal attention and supporting components |
The modules map onto cognitive-science concepts: predictive processing, self-models, hierarchical memory.
Key results
Matched-step comparison
| Model | Params | Val PPL | Relative gain | |---|---|---|---| | GPT-2 Small zero-shot | 124M | 37.50 | — | | E1: GPT-2 Small fine-tuned | 124M | 24.19 | -35.5% | | E2: CCT minus GT-Full | ~224M | 23.72 | -1.9% vs E1 | | RC2: Full CCT | 306M | 21.27 | -12.1% vs E1 | | GPT-2 Large zero-shot | 774M | 22.05 | — |
Honest reading: most of the improvement comes from domain fine-tuning; the pure architectural contribution is 2.92 PPL (12.1% relative) on top of matched fine-tuning.
Ablations: GT-Full is the star
Using strict retrain-from-scratch ablations (not just module disabling):
Simplicial message passing
GT-Full's key idea is the simplicial complex: rather than modeling only pairwise relations (graph edges), it models higher-order interactions—0-simplices are tokens, 1-simplices are pairs, 2-simplices are triples, and so on. Standard attention asks "what is the relation between token A and token B?"; simplicial message passing asks "what semantic structure do tokens A, B, C, D jointly form?" Group meaning is not the sum of pairwise relations.
The structure/consistency distinction
The paper's key conceptual contribution is an empirical law about category-theoretic inductive biases:
Why do consistency priors fail? The paper cites an independent theoretical result (Bosca & Ghrist, 2026): the forward pass of a feedforward ReLU network is already the unique harmonic extension of its boundary data—i.e., the network already minimizes sheaf discrepancy, so extra consistency losses are mathematically redundant.
Philosophical takeaway: intelligence may resemble a craftsman who needs richer tools rather than a prisoner who needs stricter rules. Constraints help only when tools are scarce.
Significance
1. Topology substitutes for parameters: principled structural priors can replace some parameter growth at a time when compute and energy are bottlenecks. 2. Math meets AI: one of the first strong empirical wins for category theory in language modeling. 3. Cognitive science validated (partly): module designs drawn from predictive processing and self-model theory captured real structure—though GT-Full dominates. 4. Methodology: the eval-only vs retrain-from-scratch ablation distinction is a useful tool for architecture comparisons.
Open questions
1. Scale: does the GT-Full advantage hold beyond 306M parameters? 2. Generalization: does the structure/consistency distinction transfer to other architectures and datasets? 3. Conditional dependence: does PrecisionWeightedPP's benefit depend on GT-Full's presence? A third counterfactual (CCT with GT-Full but without PP) is needed.