Overview
Field: NLP / Architectural Innovation Author: Al Kari Posted: 2026-05-30 arXiv: 2605.28864
English Abstract
The Cognitive Categorical Transformer (CCT) is a 306M-parameter architecture that augments a pretrained GPT-2 Small backbone with cognitively grounded components derived from category theory and several inspirations from cognitive science. Under a matched-step protocol on WikiText-103, CCT reaches 21.27 validation perplexity, compared with 24.19 for an identically fine-tuned GPT-2 Small baseline. A retrain-from-scratch ablation localizes 84% of the architectural improvement to GT-Full. We present the first ablation-validated evidence that simplicial message passing improves language-model perplexity at the 306M-parameter scale.
Key Findings
- Architecture: CCT augments pretrained GPT-2 Small (306M parameters) with category-theory- and cognitive-science-inspired components.
- Evaluation protocol: Matched-step fine-tuning on WikiText-103 over 215,000 steps, with identical data, optimizer, and learning rate between CCT and the baseline.
- Headline result: CCT reaches 21.27 validation perplexity vs. 24.19 for the identically fine-tuned GPT-2 Small baseline.
- Attribution: The architecture contributes a 2.92 PPL improvement (≈12% relative reduction).
- Ablation (retrain-from-scratch): 84% of the architectural gain (2.45 / 2.92 PPL) is localized to the GT-Full simplicial message-passing component.
- Scale of evidence: First ablation-validated demonstration that simplicial message passing improves language-model perplexity at the 306M-parameter scale.
- Empirical pattern — structure/consistency distinction: Category-theoretic priors that introduce new topology improve language modeling, whereas priors that enforce coherence/coherence identities do not.