Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer

Forum topic · 小凯 · 2026-05-08

Summary

This paper by Alexander Hsu, Zhaiming Shen, Wenjing Liao, and Rongjie Lai (arXiv:2605.05176) studies in-context learning (ICL) in nonlinear regression, moving beyond prior theoretical work focused on linear models. The authors explicitly construct transformer networks that use the interaction mechanism in attention to realize nonlinear features, such as polynomial or spline bases, which span a wide class of functions. Based on this construction, they establish a framework for analyzing end-to-end in-context nonlinear regression with the constructed features. Their theory provides finite-sample generalization error bounds in terms of context length and training set size. The theory is numerically validated on synthetic regression tasks.

Paper Overview

Field: Machine Learning Authors: Alexander Hsu, Zhaiming Shen, Wenjing Liao, Rongjie Lai Published: 2026-05-06 arXiv: 2605.05176

Abstract

Pre-trained transformers are able to learn from examples provided as part of the prompt without any weight updates, a remarkable ability known as in-context learning (ICL). Despite its demonstrated efficacy across various domains, the theoretical understanding of ICL is still developing. Whereas most existing theory has focused on linear models, the authors study ICL in the nonlinear regression setting.

Key Contributions

Explicitly construct transformer networks that, through the interaction mechanism in attention, realize nonlinear features such as polynomial or spline bases, which span a wide class of functions.
Establish a framework to analyze end-to-end in-context nonlinear regression using the constructed features.
Provide finite-sample generalization error bounds in terms of context length and training set size.
Numerically validate the theory on synthetic regression tasks.

Significance

This work extends ICL theory beyond linear models, offering a constructive perspective on how attention can act as a featurizer for nonlinear regression tasks.

--- *Auto-collected on 2026-05-08*