SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with LLMs

Forum topic · 小凯 · 2026-09-28

Summary

SemMSA is a latent semantic-aided framework for Multimodal Sentiment Analysis (MSA) that addresses spurious generation and noisy guidance in incomplete multimodal data. Proposed by Wenhao Li, Zhibin Wu, Chong Xiao, and Qiangchang Wang (arXiv:2609.30238), the method uses LLMs to build rich sentiment-relevant semantics and integrates them with all modalities via anchor-free spectral alignment. It comprises two components: Cross-modal Semantic Refinement (CSR), which adapts visual and acoustic representations into a frozen LLM embedding space and iteratively refines continuous discriminative semantic states without decoding explicit text; and Cross-modal Spectral Alignment (CSA), which aligns refined semantics with all modalities by enhancing dominant spectral components of per-modality kernel Gram matrices, capturing global nonlinear dependencies without a predefined anchor modality. Instance-level spectral separation preserves cross-sample discriminability and mitigates representation collapse. Experiments on SIMS, MOSI, and MOSEI benchmarks show state-of-the-art performance.

Paper Overview

Field: NLP Authors: Wenhao Li, Zhibin Wu, Chong Xiao, Qiangchang Wang Published: 2026-09-24 arXiv: 2609.30238

Abstract (translated)

Recent research on Multimodal Sentiment Analysis (MSA) has focused on learning from language, visual, and acoustic modalities with incomplete data to infer human sentiment. Most studies typically compensate for missing information by reconstructing modality features or designing complicated fusion mechanisms. However, these methods still suffer from spurious generation and noisy guidance due to the lack of high-level semantic grounding in partially observed multimodal evidence.

To address these issues, the authors propose SemMSA, a latent semantic-aided framework that constructs rich sentiment-relevant semantics with LLMs, fully integrating with all modalities via anchor-free spectral alignment. The framework consists of two main components:

Cross-modal Semantic Refinement (CSR)

Adaptively extracts visual and acoustic representations through corresponding adapters, forming a unified multimodal prefix with language in a frozen LLM embedding space.
Iteratively generates continuous discriminative semantic states via a token-efficient latent refinement process, without decoding explicit text.

Cross-modal Spectral Alignment (CSA)

Aligns the refined semantics with all modalities simultaneously by enhancing the dominant spectral components of each modality's kernel Gram matrix.
Captures global nonlinear dependencies across all representations without relying on a predefined anchor modality.

Additionally, an instance-level spectral separation constraint preserves discriminability across samples and mitigates representation collapse.

Results

Extensive experiments on the SIMS, MOSI, and MOSEI benchmarks demonstrate that SemMSA achieves state-of-the-art performance.

--- *Auto-collected on 2026-09-28. Source: arXiv:2609.30238*