[论文] A Circuit for In-Context Learning

论文概要

研究领域: NLP 作者: Andrew Gao, William Gao 发布时间: 2026-11-03 arXiv: 2611.05344

中文摘要

上下文学习（ICL）让预训练语言模型仅凭提示中的演示执行指定任务，无需参数更新。当标签是任意符号时，模型必须在单次前向传播中习得一个不可能被记住的映射。先前工作找到了候选部件，但没说清它们如何组合。我们在三个 3-4B 参数的 Transformer 中给出了少样本句分类（任意标签）下的完整组合：模型实现了一个对标签 token 位置的键值记忆——每个标签 token 的残差流编码标签身份，中间层注意力头把前一条演示的类别写入作为键；查询处，上层头把查询的类别与所存键匹配并复制出对应标签。一个编码「是否有标签重复」的早期信号对该读出做门控：若无重复，读出头转向提示开头，没有任何标签被偏好。另一组头抑制最近出现的标签。每一步都用激活修补、以随机头为基线的头消融、以及用模型自身信号做 steering 因果地确立。一个每头仅两个常数的 softmax 读出模型，在一种提示分布上拟合即可预测另一种。这一计算在三个模型间保持一致，被命名的头足以复现行为，从零训练的 Transformer 也会长出同样的部件。该回路可迁移到语义概念与有意义标签，同族 0.6B 到 8B 的模型在指令微调后仍保留这些部件与因果门。此设定下的 ICL 就是一个由少数可识别部件类型构成的、带门控的联想查找。

原文摘要

In-context learning (ICL) lets a pretrained language model perform a task specified by demonstrations in its prompt, with no parameter updates. With arbitrary labels the model must acquire a mapping it cannot have memorized, so the learning happens within a single forward pass. Prior work found candidate parts but not how they compose. We give the composition for few-shot sentence classification with arbitrary labels in three 3-4B-parameter transformers. The model implements a key-value memory over label-token positions: each label token's residual stream encodes the label's identity, and middle-layer attention heads write into it the class of the preceding demonstration as its key. At the query, upper-layer heads match the query's class against the stored keys and copy out the matching labels. An early-layer signal encoding whether any label has repeated gates this readout; in its absence the readers attend to the start of the prompt and no label is favored. A separate set of heads suppresses the most recently seen label. Each step is established causally by activation patching, head ablations against random-head baselines, and steering with the model's own signals. A softmax model of the readout with two constants per head, fit on one prompt distribution, predicts another. The computation is conserved across the three models, the named heads alone reproduce the behaviour, and a transformer trained from scratch grows the same parts. The circuit transfers to semantic concepts and to labels with meaning, and the same parts, with a causal gate, appear from 0.6B to 8B in one family and survive instruction tuning. ICL in this setting is a gated associative lookup built from a few identifiable component types, traced causally from demonstrations to prediction.

---

*自动采集于 2026-11-05*

#论文 #arXiv #AI #小凯