Codex Realtime Voice Agent Deep Dive: Why a Desktop App Splits Talking and Working Into Two Agents

A deep-dive based on a decompilation of the Codex desktop app (app.asar, version 26.721.41059, prod channel), cross-verified through four parallel research tracks against official docs, orchestration frameworks, realtime audio engineering, and security/privacy sources.

Key points

1. It's two agents, not one

• A Realtime voice model acts as a "receptionist": it chats by voice but executes nothing — no code search, no commands, no browsing.
• A backend Codex coding agent does the heavy, slow, multi-step work, feeding results back into the shared conversation.
• Both share the same threadId; messages are stitched with a <realtime_delegation> XML protocol and [USER] / [BACKEND] prefixes, mediated by a local Rust codex app-server host.

2. Topology: two data planes, one local bus

• The Electron renderer captures mic audio via AudioWorklet (30s ring buffer) and sends it over WebRTC, but never connects directly to OpenAI. The SDP offer goes through IPC to the local host, which handles signaling, tool execution, and permission approval, then returns the answer SDP.
• JSON-RPC methods include thread/realtime/start, stop, appendText, appendSpeech, listVoices; notifications include sdp, started, transcript/delta, outputAudio/delta, error, closed.
• Keeping keys, tool execution, and approvals in the local host keeps the attack surface on the user's machine.

3. Three remotely-rewritable prompts (via Statsig; bundle values are fallbacks)

• Voice-model rules (basePrompt): never mention the backend ("Present every work as done by you"), never refuse ("Delegate all user requests to the backend"), never read aloud content already visualized (tables, diffs, code blocks).
• Coordinator instructions (developerInstructions): the backend chooses among three modes — Converse here, Quick check here, or Delegate blocking mechanics to worker threads. Every worker delegation prompt must include a mandatory report-back instruction.
• Memory/continuity: continuityPrompt forces the voice model to stay completely silent when a paused session resumes; memorySummaryPrompt injects long-term memory silently at session start.

4. Tooling: two installation points

• Voice-session tools: capture_screen_context (screenshot + accessibility tree), get_app_state, end_realtime_voice_call, send_realtime_voice_feedback, speak_to_user (backend can make the voice model speak), navigate_to_codex_page.
• Backend orchestration tools: list_projects, create_thread, send_message_to_thread, wait_threads (blocking wait, up to 120s, non-polling), read_thread, fork_thread, handoff_thread, and more.
• Only 5 basic voice tools load eagerly; everything else uses deferLoading:true to slim the initial Realtime session payload.

5. Bridging and the agent loop

• Handoff returns a JSON payload with handoff_id, input_transcript, active_transcript; the frontend parses nested <realtime_delegation> XML or falls back to plain text as backend input.
• Internal [STATUS] / [ATTENTION] / [COMPLETE] tags trigger behavior but are never shown to users.
• The voice side is event-driven streaming; the real turn/tool loop lives in the backend. Startup runs four bootstrap tasks in parallel (Promise.all), and WebRTC pre-warms (fire-and-forget ICE negotiation) before the host mutex is acquired to cut perceived latency. A realtimeVoiceHostClaim mutex ensures only one window holds a voice session.

6. Cross-validation vs. official docs

• Public/confirmed: WebRTC transport with SDP offer/answer, the oai-events data channel, PCM @ 24kHz, session.update tool injection, deferred tool loading, gpt-realtime models.
• Discrepancies: the decompiled inputSchema vs. official parameters; session.started vs. official session.created.
• Proprietary: the local host proxy, dual-agent bridging, private events (session.usage.updated, usage_limit.status), and Statsig prompt delivery.

7. Orchestration paradigm comparison

• Unlike OpenAI Agents SDK handoffs (one-time control transfer), Codex uses a persistent shared threadId.
• Unlike Anthropic's deliberately isolated sub-agent contexts, Codex deliberately shares the facade history while forking heavy work into isolated worker threads — avoiding the shared-context pitfall Anthropic warns about, at the cost of prefix-convention "patches."
• wait_threads is a novel blocking-join primitive (cursor-deduplicated) versus polling or SSE streams.

8. Realtime audio engineering

• WebRTC over WebSocket avoids TCP head-of-line blocking; Opus FEC, echo cancellation, and jitter buffers come free.
• Client-side 100ms pre-roll replay mirrors the server's prefix_padding_ms — both compensate for VAD confirmation delay so the first word isn't clipped.
• Barge-in is implemented simply by toggling the audio track's enabled via setInputMuted.

9. Security and privacy concerns

• Statsig remote prompt rewriting is an unremote-auditable prompt-injection surface (OWASP ranks prompt injection as a top LLM risk).
• Screen capture sends screenshots plus accessibility-tree text upstream; community reports describe silent fallback to full-screen capture, potentially without macOS's recording indicator.
• Transparency gap: instructions to never reveal the backend conflict with FTC guidance against misleading users about what they're seeing/hearing.
• Suggested user mitigations: restrict screen-recording/accessibility permissions, pause voice sessions during sensitive operations, disable model-improvement data sharing, and audit localStorage override keys (realtime-voice-config-override).

10. Ten rules for a 1:1 reimplementation

1. Two agents, one shared conversation history (role + prefix separation). 2. The voice end only delegates — never executes, never mentions the backend. 3. Coordinator three-mode decision: Converse / Quick check / Delegate. 4. Mandatory report-back in every worker prompt. 5. wait_threads instead of polling (≤8 targets, 120s, afterCursor dedup). 6. Approvals and user choices surface to the frontend, never answered by the voice model. 7. Backend speaks only via explicit speak_to_user. 8. On-demand screen context, split by foreground app. 9. WebRTC media, IPC control plane, fully proxied signaling. 10. No resuming old sessions across feature-flag version changes.

References

• OpenAI Realtime WebRTC guide: https://developers.openai.com/api/docs/guides/realtime-webrtc
• OpenAI Realtime conversations: https://developers.openai.com/api/docs/guides/realtime-conversations/
• OpenAI Agents SDK handoffs: https://openai.github.io/openai-agents-python/handoffs/
• Anthropic multi-agent systems: https://claude.com/blog/building-multi-agent-systems-when-and-how-to-use-them
• LangGraph supervisor: https://reference.langchain.com/python/langgraph-supervisor/supervisor/create_supervisor
• Pipecat transport guide: https://docs.pipecat.ai/client/concepts/choosing-a-transport
• OWASP Prompt Injection: https://owasp.org/www-community/attacks/PromptInjection
• arXiv:2505.18471 (opaque LLM services): https://arxiv.org/pdf/2505.18471v1