Paper Overview
Field: NLP / Architecture Innovation Author: Al Kari Published: 2026-05-30 arXiv: 2605.28864
Summary
The Cognitive Categorical Transformer (CCT) is a 306M-parameter architecture that augments a pretrained GPT-2 Small backbone with cognitively grounded components derived from category theory and several inspirations from cognitive science.
Under a matched-step protocol on WikiText-103 (215,000 steps, with matched data, optimizer, and learning rate), CCT reaches 21.27 validation perplexity, compared with 24.19 for an identically fine-tuned GPT-2 Small baseline. This means the architecture itself contributes an improvement of 2.92 PPL (~12% relative).
Ablation experiments show that the GT-Full simplicial message passing mechanism accounts for 84% of the architectural improvement (2.45 / 2.92 PPL). This is the first evidence validating that simplicial message passing improves language-model perplexity at the 306M-parameter scale.
The paper also proposes an empirical structure/coherence distinction: category-theoretic priors that add new topology improve language modeling, while priors that enforce coherence identities do not.
Original Abstract
The Cognitive Categorical Transformer (CCT) is a 306M-parameter architecture that augments a pretrained GPT-2 Small backbone with cognitively grounded components derived from category theory and several inspirations from cognitive science. Under a matched-step protocol on WikiText-103, CCT reaches 21.27 validation perplexity, compared with 24.19 for an identically fine-tuned GPT-2 Small baseline. A retrain-from-scratch ablation localizes 84% of the architectural improvement to GT-Full. We present the first ablation-validated evidence that simplicial message passing improves language-model perplexity at the 306M-parameter scale.