Working Memory Constraints Scaffold Learning in Transformers under Data Scarcity

Forum topic · 小凯 · 2026-04-24

Summary

A forum post introduces an arXiv paper (2604.20789) by Pranava Madhyastha and Dagmar Adamcova investigating how human-like working memory constraints can be integrated into the Transformer architecture. The authors implement several cognitively inspired attention variants, including fixed-width window attention and temporal-decay-based attention, and train modified GPT-2 models from scratch on developmentally plausible datasets of 10M and 100M words. Performance is evaluated on grammatical judgment tasks using BLiMP and on alignment with human reading time data. Results show that these constraints, particularly fixed-width attention, significantly improve grammatical accuracy, especially when training data is scarce. The constrained models also tend to show stronger alignment with human processing metrics, suggesting that working memory limits act as beneficial inductive biases that guide models toward more robust language representations in low-data regimes.

Paper Overview

Field: NLP Authors: Pranava Madhyastha, Dagmar Adamcova Published: 2026-04-22 arXiv: 2604.20789

Abstract

We investigate the integration of human-like working memory constraints into the Transformer architecture and implement several cognitively inspired attention variants, including fixed-width windows based and temporal decay based attention mechanisms. Our modified GPT-2 models are trained from scratch on developmentally plausible datasets (10M and 100M words). Performance is evaluated on grammatical judgment tasks (BLiMP) and alignment with human reading time data. Our results indicate that these cognitively-inspired constraints, particularly fixed-width attention, can significantly improve grammatical accuracy especially when training data is scarce. These constrained models also tend to show a stronger alignment with human processing metrics. The findings suggest that such constraints may act as beneficial inductive biases, guiding models toward more robust linguistic representations, particularly in data-limited settings.

Key Points

Working memory constraints are incorporated into Transformer attention mechanisms.
Two cognitively inspired variants: fixed-width window attention and temporal-decay-based attention.
Modified GPT-2 models trained from scratch on 10M and 100M word corpora (developmentally plausible sizes).
Evaluated via BLiMP grammatical judgment tasks and alignment with human reading time data.
Fixed-width attention notably improves grammatical accuracy when training data is scarce.
Constrained models show stronger alignment with human processing metrics, suggesting such constraints serve as useful inductive biases.