DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging

Forum topic · 小凯 · 2026-09-01

Summary

DARTS (Decoder-Aware Representation Tuning via Surgery) is a new method from researchers Aaryan Ajay Sharma, Sai Nishanth Padala, and Seganrasan Subramanian addressing representation bias in merged decoder-only LLMs. Model merging combines multiple task-specific fine-tuned LLMs into a single multi-task model without extra training, but merged models exhibit systematic drift between their hidden states and those of the source models. Prior correction techniques were designed for encoder-based vision models and do not transfer to decoders, where two decoder-specific challenges arise: (1) the causal attention mask causes bias to accumulate across token positions, requiring position-dependent correction, and (2) high-entropy decision-critical positions matter far more than low-entropy ones. DARTS combines an entropy-weighted L1 loss, weighting corrections at high-entropy positions, with per-position additive bias terms that capture position-dependent error without over-parameterization. Evaluated on Llama-2-7B across code generation (HumanEval), math reasoning (GSM8K), and instruction following (AlpacaEval), DARTS significantly improves over standard surgery methods while adding only ~0.1% extra parameters. Paper: arXiv:2608.28547.

Paper Overview

Field: Machine Learning Authors: Aaryan Ajay Sharma, Sai Nishanth Padala, Seganrasan Subramanian Published: 2026-08-28 arXiv: 2608.28547

Abstract

Model merging combines multiple task-specific fine-tuned LLMs into a single multi-task model without additional training. However, merged models are known to suffer from representation bias: systematic drift between the merged model's hidden states and those of each individual source model. Prior work (Yang et al., 2024a) studies and mitigates this bias for encoder-based vision models using a lightweight correction module trained with L1 loss. However, such bias has not been studied for decoder models due to their autoregressive nature.

The authors analyze representation bias in decoder models and identify two challenges absent in encoders:

1. Bias accumulation across positions: the causal attention mask causes bias to accumulate across token positions, requiring position-dependent correction. 2. Non-uniform token importance: not all token positions are equally important — high-entropy (decision-critical) positions matter far more than low-entropy positions.

The DARTS Method

DARTS (Decoder-Aware Representation Tuning via Surgery) addresses both challenges with:

A novel entropy-weighted L1 loss that weights corrections at high-entropy positions where generation quality is most affected.
Per-position additive bias terms that capture position-dependent errors without over-parameterization.

Evaluation

DARTS was evaluated on Llama-2-7B across three domains:

Code generation (HumanEval)
Math reasoning (GSM8K)
Instruction following (AlpacaEval)

Results show DARTS achieves significant improvements over standard surgery methods while adding only negligible parameters (~0.1% of total parameters).

---

*Auto-collected on 2026-09-01*