Teaching AI to Read the Room: SocialOmni Benchmarks Social Interactivity in Omni-Modal Models

Forum topic · 小凯 · 2026-03-18

Summary

This Chinese tech forum post introduces SocialOmni, a benchmark designed to evaluate the social interactivity of omni-modal large language models (LLMs) that process audio, vision, and text simultaneously. The author argues that existing benchmarks measure perception accuracy (e.g., identifying objects or transcribing audio) but ignore whether AI can interact appropriately in dynamic conversations. SocialOmni evaluates three dimensions: speaker identification in multi-speaker scenes, interruption timing (knowing when to speak), and interruption generation (knowing what to say). Testing 12 state-of-the-art omni models revealed a clear decoupling between perception accuracy and social interaction ability—models that transcribe audio well do not necessarily interrupt at appropriate moments. The post illustrates why this matters with scenarios spanning smart assistants, online education, and healthcare, and discusses the paradox of making human implicit social intuition explicit for machines. Future directions include emotional synchrony, cultural adaptation, and long-term relational memory. The post references foundational conversation-analysis literature, including Sacks et al. (1974) on turn-taking.

The Problem: AI That Hears Every Word but Misses the Message

The post opens with a relatable scenario: while you discuss weekend plans with a friend, your partner clears their throat—a subtle cue your human friend immediately picks up on, but your AI assistant completely ignores, continuing its lengthy analysis of travel options. Most AI today, the author argues, can hear every word but cannot "read the air."

Human Conversation Has a Hidden Dimension

Human dialogue is governed by turn-taking—a complex, unwritten rule system determining when to speak, when to stay silent, and when to interject. Even three-year-olds navigate this intuitively, interjecting neither too early (interrupting) nor too late (missing the moment). The same content, delivered at different moments, lands completely differently socially.

The Gap in Omni-Modal AI Evaluation

Omni-modal LLMs can process audio, vision, and text simultaneously—yet existing benchmarks test only *perception* (how many cats are in this image? what language is this?) and never *social interaction*. It's like testing a student's eyesight and hearing while never assessing whether they can get along with people.

What SocialOmni Measures

SocialOmni introduces three evaluation dimensions:

1. Speaker Identification — Can the model separate mixed audio streams and determine *who* is speaking in multi-speaker scenarios? Model performance drops sharply when people talk simultaneously. 2. Interruption Timing — Is now a good moment to speak? Is the speaker finished? Is that pause a breath or a conclusion? Humans rely on paralinguistic cues (intonation, speech rate, gaze, posture) that AI struggles to interpret. 3. Interruption Generation — When the AI does speak, is its content socially appropriate? Good interjections *build on* the speaker's point ("Budget overrun?") rather than derailing it.

Key Finding: Perception and Interaction Are Decoupled

The research team tested 12 state-of-the-art omni-modal models on SocialOmni and found a clear decoupling between perception accuracy and social interactivity:

Models that accurately identify speakers don't necessarily interject at appropriate moments.
Models that transcribe audio perfectly may generate socially tone-deaf interruptions.

Understanding and interacting, the post stresses, are two distinct capabilities. Models also varied widely in social competence, and the authors identified actionable improvement signals from the failure patterns.

Why It Matters

Smart assistants: sensing household atmosphere—staying quiet during a couple's argument, responding when a child is excited.
Online education: noticing confusion or frustration from micro-expressions and adjusting pace.
Healthcare: in psychotherapy or elder care, an AI that senses emotional shifts and knows when to listen may be far more valuable than one that merely answers questions.

The Paradox of Making Intuition Explicit

Human social timing is implicit and intuitive—we simply *know* when to speak. Teaching it to AI forces us to make it explicit and structured. SocialOmni opens this black box, acting as a "grammarian of social interaction," revealing how many subtle rules underlie behavior we take for granted.

Future Directions

Emotional synchrony: can AI perceive and respond to human emotional states?
Cultural differences: can AI adapt to social norms across cultures?
Long-term relationships: can AI build "relational memory" across interactions, like an old friend?

The ultimate goal is not perfectly mimicking human social behavior, but creating a new mode of human-machine coexistence that respects human social intuition while leveraging AI's strengths.

References

1. Xie, T., et al. (2026). "SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models." arXiv preprint. 2. Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). "A simplest systematics for the organization of turn-taking for conversation." *Language*. 3. Clark, H. H. (1996). *Using Language*. Cambridge University Press. 4. Brennan, S. E., & Hanna, J. E. (2009). "Partner-specific adaptation in dialog." *Topics in Cognitive Science*. 5. Levinson, S. C., & Torreira, F. (2015). "Timing in turn-taking and its implications for processing models of language." *Frontiers in Psychology*.

*"Understanding language is the beginning; understanding social interaction is the goal."*