Word-level Emotional
Intensity Control in TTS
via Emotion Residual Vectors

Abstract

Modern emotional text-to-speech (TTS) models typically rely on utterance-level conditioning. However, natural emotional speech requires fine-grained variations across words to achieve nuanced, human-like expressiveness. To address this, we introduce emotion residual vectors (ERVs), which capture neutral-to-emotional deviations in word-aligned self-supervised speech embeddings as an annotation-free cue for local prosody. To stabilize intensity control, we compress ERVs into a low-dimensional bottleneck space and train a RoBERTa-based predictor to infer the projected vectors from text and emotion. During inference, the predicted vectors are converted into additive hidden-state offsets and injected into a neutral-conditioned TTS model, enabling continuous word-level intensity control. Experimental results show that our method improves local emotional controllability while preserving naturalness compared to controllable emotional TTS baselines.

Emotional text-to-speech / Emotion intensity / Word-level emotion control