In-Place Test-Time Training: A Plug-and-Play Framework for Test-Time Adaptation of LLMs

Forum topic · 小凯 · 2026-04-09

Summary

A paper on arXiv (2504.06263) by Guhao Feng, Shengjie Luo, and Kai Hua, published April 8, 2025, introduces In-Place Test-Time Training (In-Place TTT), a framework that equips large language models with test-time training capability. The static 'train then deploy' paradigm prevents LLMs from adapting weights to new information at inference time. While test-time training (TTT) offers an alternative by updating fast weights during inference, its adoption has been limited by architectural incompatibility, computational inefficiency, and misaligned fast-weight objectives for language modeling. In-Place TTT addresses these issues by treating the final projection matrix of ubiquitous MLP blocks as adaptable fast weights, enabling plug-and-play enhancement without costly retraining from scratch. Combined with a theoretically grounded objective tailored to next-token prediction in autoregressive language modeling and an efficient block-level update mechanism, the method yields a highly scalable algorithm compatible with context parallelism.

Paper Overview

Research Area: NLP Authors: Guhao Feng, Shengjie Luo, Kai Hua Published: 2025-04-08 arXiv: 2504.06263

Abstract

The static "train then deploy" paradigm fundamentally limits Large Language Models (LLMs) from dynamically adapting their weights in response to continuous streams of new information inherent in real-world tasks. Test-Time Training (TTT) offers a compelling alternative by updating a subset of model parameters (fast weights) at inference time, yet its potential in the current LLM ecosystem is hindered by critical barriers including architectural incompatibility, computational inefficiency and misaligned fast weight objectives for language modeling. In this work, we introduce In-Place Test-Time Training (In-Place TTT), a framework that seamlessly endows LLMs with Test-Time Training ability. In-Place TTT treats the final projection matrix of the ubiquitous MLP blocks as its adaptable fast weights, achieving a "plug-and-play" enhancement of LLMs without expensive retraining from scratch. Combined with a theoretically grounded objective tailored to the next-token prediction task in autoregressive language modeling, along with an efficient block-level update mechanism, the method forms a highly scalable algorithm compatible with context parallelism.

Key Points

Problem: LLMs cannot adapt weights after deployment, limiting their response to new information streams.
Approach: Use the final projection matrix of MLP blocks as fast weights updated at inference time — no architectural overhaul or retraining required.
Objective: A theoretically motivated fast-weight objective aligned with next-token prediction for autoregressive language modeling.
Efficiency: Block-level update mechanism makes the method scalable and compatible with context parallelism.

---

*Auto-collected on 2026-04-09.*