EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation

Forum topic · 小凯 · 2026-10-01

Summary

EmoRES-TTS (arXiv:2609.38157) introduces a training-free vector steering method for improving emotion controllability in text-to-speech models. The authors discover that emotion vectors can be decomposed into a shared component that pushes speech away from neutral expression and a residual component that directs generation toward the requested emotion. Building on this insight, EmoRES controls both components separately without retraining the frozen backbone model. On IEMOCAP, EmoRES outperforms the prior CoCoEmo method across four objective emotion metrics on IndexTTS-2 and CosyVoice2 backbones, with rank correlation gains of 26.13 and 12.97 percentage points (118.8% and 33.1% relative) and emotion hit-rate gains of 12.95 and 6.92 percentage points. Human evaluations show up to 35.0% relative improvement in recognizing the requested dominant emotion, 17.3% fidelity improvement, and preference for EmoRES naturalness in up to 63.8% of pairwise comparisons.

Overview

Field: NLP Authors: Kuan-Po Huang, Haohe Liu, Puyuan Peng, Haibin Wu, Zhaoheng Ni, Hung-yi Lee, Jinwon Lee, Neha Chachra Published: 2026-09-29 arXiv: 2609.38157

Abstract

Emotion-conditioned text-to-speech (TTS) models may fail to express the requested emotion reliably, and improving controllability by additional training is costly in both computation and emotion-labeled speech training data. The authors therefore study vector steering, a training-free approach that modifies the internal representations of a frozen model.

CoCoEmo, a conventional vector steering method for emotion TTS, treats each emotion vector as an indivisible direction controlled by a single global strength, limiting adherence to the requested emotion.

Key Idea

The paper first discovers that an emotion vector can be decomposed into two parts:

A shared component that moves speech away from neutral expression
A residual component that directs generation toward the requested emotion

Building on this finding, EmoRES (Emotion Residual-Enhanced Steering) controls the two components separately without retraining the backbone model.

Results

Evaluated on IEMOCAP, EmoRES outperforms CoCoEmo on all four objective emotion metrics across two backbones (IndexTTS-2 and CosyVoice2):

Rank correlation: +26.13 and +12.97 percentage points (118.8% and 33.1% relative gains)
Emotion hit rate: +12.95 and +6.92 percentage points (20.1% and 9.8% relative gains)

Human evaluation results:

Up to 35.0% relative improvement in listeners' recognition of the dominant requested emotion
Up to 17.3% improvement in fidelity
EmoRES preferred in up to 63.8% of pairwise naturalness comparisons

Component ablations further show that effective control requires both preserving the shared component and enhancing the residual component of the emotional steering vector.

--- *Auto-collected on 2026-10-01*