READ: New LoRA Skills Should Read but Never Write

Forum topic · 小凯 · 2026-09-29

Summary

A new paper (arXiv 2609.31600) proposes READ (Read-only Expansion of Adapter Deltas), a method for composing multiple independently trained LoRA adapters into a single model without interference or retraining. The authors trace prior failures to two implicit choices: which of the infinitely many equivalent factorizations of a LoRA update is used, and the direction of coupling between old and new skills. READ fixes both: each adapter is rewritten into a balanced canonical form that exactly preserves its update, and coupling grows only in one direction—new skills may read the input subspaces of old skills but never write into their output subspaces. Only the new skill's row in the coupling matrix is trained per addition, and merged updates fold directly into the base weights, adding no inference overhead, routing, or task-specific rules. Across four benchmark suites and two model families with sequential skill addition, READ raises average scores over the strongest published baselines—by more than 20 points on SuperGLUE and over 7 points on domain suites—with nearly all sequential stacking runs ending above all direct baselines.

Paper Overview

Field: Machine Learning Authors: Zeyan Li, Panqi Yang, Qirong Guo, Shengda Zhuo, Siyuan Qiu, Hu Xu, Chun Li, Jianfeng Xu Posted: 2026-09-25 arXiv: 2609.31600

Problem

Low-rank adapters (LoRA) make it cheap to fine-tune a large language model once per task, but combining several independently trained adapters into one model remains difficult:

Merging the updates in weight space causes interference.
Retraining on all task data is expensive.
Routing between separate adapters gives up the goal of a single combined model.

Key Idea

The paper traces the difficulty to two choices every composition method makes implicitly:

1. Factorization choice: A LoRA update admits infinitely many equivalent factorizations. The choice is invisible while an adapter serves alone, but it determines what a learned interaction between adapters can "see". 2. Coupling direction: Coupling between old and new skills can point in either direction, and the direction decides whether old skills keep computing what they originally computed.

Method: READ

READ (Read-only Expansion of Adapter Deltas) fixes both choices simultaneously:

Each adapter is rewritten into a balanced canonical form that exactly preserves its update.
Coupling grows one-way only: new skills can read the input subspaces of old skills but cannot write into their output subspaces.
When appending a new skill, the only trainable parameters are that skill's row in the coupling matrix.
Merged updates fold directly into the base weights — no inference overhead, no routing, no task-specific rules.

Results

Evaluated on four benchmark suites and two model families, adding one skill at a time:

READ lifts average scores on every suite above the strongest published baselines built from the same adapters.
SuperGLUE: improvement of more than 20 points.
Domain suites: improvement of more than 7 points.
Almost all complete sequential-stacking runs end above all direct baselines.

Takeaway

The decomposition coordinates and coupling direction that a single adapter never exposes are exactly what determine whether composed skills survive.

--- *Auto-collected 2026-09-29*