Key points
- Two agents, one shared history: The Live Agent runs a Realtime voice model (the "operator" that talks but never executes tools) and a backend Codex coding agent (the "worker" that runs commands, browses, and implements). Both share the same
threadId, distinguished by[USER]/[BACKEND]prefixes and<realtime_delegation>XML bridging. - Architecture: Electron renderer captures mic audio (AudioWorklet, 30s ring buffer) and sends it over WebRTC to the OpenAI Realtime API — but the frontend never connects directly to OpenAI. A local Rust
codexapp-server host proxies signaling (JSON-RPC over IPC), holds API keys, executes tools, and handles approvals. Media plane is WebRTC end-to-end; control plane is local IPC. - Three remotely-rewritable prompts (delivered via Statsig, with bundle fallbacks): the operator's base prompt (never mention the backend, never refuse, don't read out visualizations), the backend coordinator's developer instructions (three modes: converse / quick-check / delegate to worker threads, with mandatory report-back clauses), plus continuity and memory-summary prompts that force silence on resume.
- Tool split: 5 basic voice tools load eagerly (
capture_screen_context,get_app_state,end_realtime_voice_call,send_realtime_voice_feedback,speak_to_user); backend orchestration tools (list_projects,create_thread,send_message_to_thread,wait_threads, etc.) usedeferLoading:trueto slim the initial session payload. - Blocking join instead of polling:
wait_threadsblocks up to 120s withafterCursordeduplication — a novel alternative to polling loops and SSE streams. - Latency engineering: four-way parallel bootstrap (voices, WebRTC, history, memory summary), a silent WebRTC pre-warm before acquiring the host mutex, client-side 100ms pre-roll replay mirroring server-side
prefix_padding_ms, and a host mutex ensuring only one window owns a voice session. - Public: WebRTC transport with SDP offer/answer,
oai-eventsdata channel, 24kHz PCM, tool injection viasession.update,deferLoading,gpt-realtimemodels. - Discrepant: the decompiled
inputSchemavs. officialparameters;session.startedvs. officialsession.created. - Proprietary: the local Rust host bus, dual-agent bridge protocol, and Statsig prompt hot-swapping.
- Unlike OpenAI Agents SDK handoffs (one-time control transfer), Codex keeps a persistent shared threadId.
- Unlike Anthropic's deliberately isolated orchestrator-subagent contexts, Codex shares history — but only the facade; heavy work forks to isolated worker threads, sidestepping Anthropic's warning about shared-context domains.
- Similar in spirit to AutoGen group chat (coordinator + workers) and LangGraph supervisors, but with an explicit blocking join primitive.
- OpenAI Realtime WebRTC:
https://developers.openai.com/api/docs/guides/realtime-webrtc - OpenAI Realtime conversations:
https://developers.openai.com/api/docs/guides/realtime-conversations/ - OpenAI Agents SDK handoffs:
https://openai.github.io/openai-agents-python/handoffs/ - Anthropic multi-agent systems:
https://claude.com/blog/building-multi-agent-systems-when-and-how-to-use-them - LangGraph supervisor:
https://reference.langchain.com/python/langgraph-supervisor/supervisor/create_supervisor - Pipecat transport guide:
https://docs.pipecat.ai/client/concepts/choosing-a-transport - OWASP Prompt Injection:
https://owasp.org/www-community/attacks/PromptInjection - arXiv:2505.18471 (opaque LLM services):
https://arxiv.org/pdf/2505.18471v1
Public vs. proprietary (cross-verified against official docs)
Orchestration paradigm comparison
Security and privacy findings
1. Remote prompt rewriting via Statsig — personalities, tools, and flags can be changed server-side without user auditability (a prompt-injection surface per OWASP).
2. Screen capture — capture_screen_context returns screenshots plus accessibility-tree text; community reports of silent fallback to full-screen capture; may not trigger macOS recording indicators.
3. Transparency gap — instructions to never reveal the backend contradict FTC guidance on not misleading users.
4. Local realtime-voice-config-override in localStorage is a tamper vector but not remote.
5. User mitigations: restrict screen-recording/accessibility permissions, pause sessions during sensitive tasks, disable data-for-training sharing, inspect localStorage override keys, and use OpenAI's deletion channels.
Replication contract (10 rules, condensed)
1. Two agents sharing one threadId with role+prefix separation.
2. Voice side is a pure operator: no execution, no backend mention, no reading visualizations.
3. Coordinator uses three-mode decisions (converse / quick-check / delegate).
4. Every worker prompt mandates a report-back.
5. Wait via wait_threads, not polling.
6. Approvals/user input pass through to the frontend.
7. Backend speech requires explicit speak_to_user.
8. Screen context on demand only.
9. Media over WebRTC, control over IPC proxy.
10. Don't resume sessions across feature-flag versions.
Why this design (editor's view)
The split is forced by physics, not aesthetics: voice models must respond instantly and never stall, while coding agents must think long and run tools. Merging them into one loop yields either choppy speech or rushed code. The clever part is limiting shared context to the user-visible facade while forking heavy work into isolated threads — user feels a single conversation, work stays isolated. The biggest concern is transparency: a hidden backend, remotely editable prompts, and silent screen capture all happen in the name of smoother UX.