Summary
This paper by Pranava Madhyastha and Dagmar Adamcova investigates integrating human-like working memory constraints into the Transformer architecture. The authors implement several cognitively inspired attention variants, including fixed-width window attention and temporal decay-based attention, applied to modified GPT-2 models trained from scratch on developmentally plausible datasets of 10M and 100M words. Performance was evaluated on grammatical judgment tasks (BLiMP) and by measuring alignment with human reading time data. The results show that cognitive constraints, particularly fixed-width attention, significantly improve grammatical accuracy when training data is scarce, and constrained models exhibit stronger alignment with human processing metrics. The findings suggest that working memory constraints can act as beneficial inductive biases, guiding models toward more robust language representations, especially in low-data regimes.
Overview
- Field: NLP
- Authors: Pranava Madhyastha, Dagmar Adamcova
- arXiv: 2604.20789
Abstract
We investigate the integration of human-like working memory constraints into the Transformer architecture and implement several cognitively inspired attention variants, including fixed-width windows based and temporal decay based attention mechanisms. Our modified GPT-2 models are trained from scratch on developmentally plausible datasets (10M and 100M words). Performance is evaluated on grammatical judgment tasks (BLiMP) and alignment with human reading time data.
Our results indicate that these cognitively-inspired constraints, particularly fixed-width attention, can significantly improve grammatical accuracy especially when training data is scarce. These constrained models also tend to show a stronger alignment with human processing metrics. The findings suggest that such constraints may act as beneficial inductive biases, steering models toward more robust linguistic representations, particularly in data-limited settings.
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177618700