“Multimodal” used to mean one model could accept more than text. OpenAI’s current portfolio points to a more useful split: interaction systems that listen and respond continuously, and production systems that create and revise media inside a workflow.
GPT-Live and the Realtime API sit on the interaction side. GPT Image 2 and Sora 2 sit on the production side. They share multimodal inputs, but their latency, state, safety, and product design are fundamentally different.
First, do not conflate GPT-Live with the Realtime API
The names are close enough to cause real architecture mistakes.
GPT-Live is the July 2026 voice architecture now powering ChatGPT Voice. At launch, OpenAI said GPT-Live-1 and GPT-Live-1 mini were rolling out in ChatGPT and planned for the API later.
The developer-facing Realtime API is already generally available, but its current model line is separate. The July API release introduced GPT-Realtime-2.1 and a mini variant. GPT-Realtime-2.1 accepts text, audio, and images; produces text and audio; supports tool calling and prompt caching; and exposes configurable reasoning effort.
| Surface | Current role | Availability described by source |
|---|---|---|
| GPT-Live-1 / mini | Full-duplex ChatGPT Voice experience | ChatGPT rollout; API access announced as future at launch |
| GPT-Realtime-2.1 / mini | Low-latency speech-to-speech developer model | Realtime API |
| GPT Live Transcribe | Low-latency streaming speech-to-text | Realtime API |
| GPT Transcribe | Accurate bounded or final-turn transcription | Audio and Realtime workflows |
This is a product boundary, not a branding detail. Do not design an API integration around a ChatGPT product capability until the developer documentation exposes it.
GPT-Live changes the conversation architecture
Traditional voice systems are turn-based: listen, detect silence, transcribe, reason, synthesize, speak. That pipeline creates awkward interruptions and makes the assistant appear absent while deeper work runs.
OpenAI says GPT-Live uses a full-duplex architecture that continuously processes input while generating output. It can decide many times per second whether to listen, speak, pause, interrupt, or use a tool. For more complex search or reasoning, the voice layer can delegate to a frontier model and keep the interaction moving while the deeper task runs.
That separation is the important design idea:
- a low-latency interaction loop manages timing, turn-taking, and conversational continuity;
- a slower reasoning loop handles research, planning, or agent work;
- the interaction loop returns and explains the result.
Voice agents built through the Realtime API can adopt a similar split even if the exact GPT-Live product model is unavailable. The Realtime guide recommends choosing among voice-agent, translation, and transcription sessions based on the actual outcome rather than forcing every audio task through one assistant loop.
Voice creates a new class of failure
Text gives a user time to inspect an output before acting. Voice is transient, interruptible, and emotionally stronger. A production design needs measures that ordinary chat tests miss:
- false interruption and failure-to-interrupt rates;
- response latency across noise, accents, and weak connections;
- exact capture of names, codes, amounts, and addresses;
- tool calls triggered by misunderstood speech;
- disclosure that the voice is synthetic;
- escalation quality in distress or high-risk situations;
- whether the system can safely stop speaking mid-output.
OpenAI’s GPT-Live announcement describes new audio-native safety evaluations, real-time intervention, predefined voices, and protections against voice impersonation. Those controls are specific to OpenAI’s product. An API developer remains responsible for workflow-level consent, recording policy, identity verification, and action approval.
GPT Image 2 turns generation into an editing workflow
GPT Image 2 is OpenAI’s current state-of-the-art image generation model, with text and image input, image output, flexible sizes, high-fidelity input handling, and inpainting.
The image generation guide exposes two integration patterns:
- use the Image API for a bounded generation or edit;
- use the Responses API image tool for conversational, multi-turn generation and editing inside a broader agent flow.
That second route matters for production. An agent can inspect references, generate an asset, receive revision instructions, and keep the visual state in the same task context. The application still needs deterministic checks for dimensions, file type, embedded text, brand policy, copyright risk, and accessibility.
The May 2026 API changelog also records the removal of DALL·E 2 and DALL·E 3 snapshots. Teams with old model IDs should treat that as a completed deprecation, not an upcoming migration.
Sora 2 makes video a job system
The Videos API exposes Sora 2 for rapid iteration and Sora 2 Pro for higher-fidelity output. The API can:
- create clips from text or a reference image;
- reuse character assets;
- extend a completed clip;
- edit a video with targeted changes;
- return supporting assets;
- submit large queues through the Batch API.
Video generation is asynchronous. A create request starts a job, after which an application polls status or consumes a webhook and downloads the completed asset. That makes video fundamentally different from streaming text: product design needs queues, progress, cancellation, retry policy, storage, and failed-render handling.
The current guide says both Sora 2 variants support 16- and 20-second generations, with Pro supporting 1080p formats. It also documents meaningful restrictions, including rejection of real people, public figures, copyrighted characters, copyrighted music, and human faces in input images under the current policy.
A selection matrix
| Job | Use first | Why |
|---|---|---|
| Natural two-way assistant conversation | Realtime voice-agent session | Low latency, audio output, state, and tools |
| Continuous interpretation | Realtime translation session | Dedicated translation lifecycle without assistant turns |
| Live captions or call transcript | GPT Live Transcribe | Streaming partial text with controllable delay |
| High-quality file transcript | GPT Transcribe or bounded transcription API | Accuracy matters more than conversational latency |
| One image or edit | Image API with GPT Image 2 | Simple bounded request |
| Iterative visual work inside an agent | Responses image-generation tool | Multi-turn context and tool composition |
| Fast video concepts | Sora 2 | Iteration speed |
| Final higher-resolution video asset | Sora 2 Pro | Fidelity and 1080p support |
A multimodal release gate
Before launch, test the artifact and the interaction separately.
Interaction checks
- latency, interruption, silence, and noise;
- language and accent coverage;
- exact entity capture;
- tool-call confirmation and approval;
- safe handoff to a person.
Artifact checks
- dimensions, duration, codec, and file integrity;
- textual accuracy inside generated media;
- reference fidelity and unwanted edits;
- rights, likeness, and policy compliance;
- provenance metadata and audit record;
- human approval before external publication.
Multimodal capability is only useful when the surrounding product knows whether it is managing a live relationship or producing a reviewable artifact. The former needs timing and escalation. The latter needs validation and provenance.