Thinking Machines Lab's Interaction Models: Full-Duplex AI That Listens While It Speaks

Forum topic · 小凯 · 2026-05-17

Summary

Thinking Machines Lab, founded by former OpenAI CTO Mira Murati, has introduced Interaction Models—a new AI architecture built for real-time, full-duplex human-machine interaction. Instead of the traditional turn-based wait-listen-respond loop, the TML-Interaction-Small model (276B-parameter MoE with 12B active parameters) divides continuous audio, video, and text streams into 200ms micro-turns, enabling the model to listen, watch, think, and speak simultaneously. A dual-model system pairs a real-time interaction model with an asynchronous background model for deep reasoning and tool use, streaming results back into the live conversation. The architecture uses encoder-free early fusion, feeding raw audio (dMel spectrograms) and video patches directly into a transformer, trained end-to-end. Serving optimizations include Streaming Sessions (open-sourced to SGLang) and gather+gemv MoE kernels. The model achieves 0.40s turn-taking latency and scores 77.8 on FD-bench v1.5, outperforming GPT-Realtime-2.0 and Gemini live models on interaction quality and proactive speech benchmarks (TimeSpeak 64.7%, CueSpeak 81.7%) where existing systems score near zero. Remaining challenges include long-session context management, network sensitivity, and scaling beyond the 'Small' tier.

Overview

Thinking Machines Lab — founded by Mira Murati, former OpenAI CTO — argues that interaction is not an add-on feature but a native property of AI models. Their new Interaction Models replace the conventional turn-based (half-duplex) voice assistant design with a full-duplex, real-time architecture.

Model: TML-Interaction-Small — 276B-parameter MoE, 12B active, 0.40s turn latency.

The Problem: Architectural "Deafness"

Current voice assistants (ChatGPT Voice, Gemini Live, Siri) operate in half-duplex mode: they use voice activity detection (VAD) to decide when the user has finished, then respond. While generating output, they cannot hear or see anything new — they cannot be interrupted meaningfully, cannot watch you code in real time, and cannot speak proactively.

Micro-Turn Architecture: 200ms Slices

The Interaction Model divides continuous input into 200ms micro-turns. Every 200ms it:

1. Receives the user's audio chunk + video frames 2. Thinks and generates 200ms of output simultaneously 3. Emits text, audio, or both

There is no "turn boundary." All streams (user audio, video frames, model output) flow concurrently.

Dual-Model System: Real-Time + Asynchronous

Interaction Model: always online, handles micro-turns, interruptions, immediate responses; delegates deep reasoning upward.
Background Model: runs asynchronously for complex reasoning, tool calls, and web search, streaming results back into the conversation.

Users never wait for the background model — the interaction model keeps chatting meanwhile. This mirrors human collaboration: "let me take a look" now, with the deep answer arriving 30 seconds later.

Technical Details

Encoder-free early fusion: no separate ASR/TTS modules. Raw audio (dMel spectrogram → lightweight embedding) and video (40×40 patches → hMLP) feed directly into the transformer; audio is decoded via a flow-matching head; everything is trained end-to-end, reducing cascade errors and latency.
Streaming Sessions: clients send HTTP requests every 200ms; the server appends chunks to persistent GPU sequences instead of reallocating memory per request. Open-sourced to SGLang.
MoE kernel optimization: standard grouped GEMM is replaced with a gather+gemv strategy suited to small-batch, low-latency bidirectional serving.
Trainer-sampler alignment: batch-invariant kernels provide bitwise-aligned outputs across parallelism strategies; consistent Split-KV partitioning between prefill and decode keeps attention deterministic.

Capabilities Existing Systems Lack

| Capability | Example | Existing systems | |---|---|---| | Simultaneous speech | Real-time translation while user keeps talking | No | | Visual proactivity | "Count my push-ups" | Audio-trigger only | | Time awareness | "Remind me to breathe every 4 seconds" | No | | Interruption awareness | Instant correction mid-utterance | Waits for user to finish | | Listen-and-search | Auto-lookup mid-conversation | Post-conversation only |

New Benchmarks

TimeSpeak: 64.7% — proactive timed speech
CueSpeak: 81.7% — instant correction on user errors
RepCount-A: 35.4% — real-time visual action counting
ProactiveVideoQA: substantially above the 25.0 baseline
Charades: mIoU 32.4 — temporal action localization

Existing models, including GPT Realtime-2.0, score near zero on these benchmarks.

Benchmark Comparison

| Benchmark | TML-Interaction-Small | GPT-Realtime-2.0 (minimal) | GPT-Realtime-1.5 | Gemini-3.1-flash-live (minimal) | |---|---|---|---|---| | FD-bench v1.5 (interaction quality) | 77.8 | 47.8 | 48.3 | 54.3 | | FD-bench v1 (turn latency) | 0.40s | 1.18s | 0.59s | 0.57s | | Audio MultiChallenge APR | 43.4% (instant) | 37.6% | 34.7% | 26.8% | | FD-bench v3 response quality | 82.8% | - | - | - |

Notably, TML-Interaction-Small is an instant model (no extended reasoning) yet beats GPT-Realtime-2.0's thinking mode on interaction quality.

Safety

Spoken refusals generated with TTS so rejections sound natural
Multi-turn refusal consistency via automated red-teaming
Harmbench refusal rate: 99.0%
Alignment of refusal boundaries between voice and text

Limitations

1. Long sessions: context accumulates quickly at 200ms granularity; proactive context management is unsolved 2. Network sensitivity: dropped frames degrade the experience severely 3. Scale: 276B MoE is the "Small" tier; larger models are too slow for real-time serving 4. Background agents: async deep reasoning + real-time interaction is "just beginning"

Implications

Short term (6–12 months): voice assistants shift from wait-and-respond to real-time conversational partners; existing products must re-architect
Mid term (1–2 years): tutoring that watches you solve problems, sales/support that senses hesitation, medical consultation combining speech and visual signs
Long term (3–5 years): the "agent dispatch" metaphor may give way to an always-present collaborator; interaction bandwidth becomes capability bandwidth

Conclusion

The Interaction Model is not a faster voice assistant — it is a shift from turn-taking to continuous presence, from half-duplex to full-duplex, from waiting for input to being continuously aware.

Reference: Thinking Machines Lab blog — https://thinkingmachines.ai/blog/interaction-models/