Skip to main content
The Inworld Realtime API uses an OpenAI Realtime API-compatible event system to facilitate voice experiences. This guide walks through configuring each layer — STT, LLM, and TTS — plus the conversation- and observability-level controls that span all three. For the field-by-field reference of Inworld extensions, see Inworld Realtime API Extensions.

​
Configure a session

For WebSocket, the connection starts with a session.created event. For WebRTC, send session.update as soon as the data channel opens. In both cases, use session.update to configure your session. Here you can set:
  • model — LLM provider and model (e.g. openai/gpt-4.1-nano) or router (e.g. inworld/latency-optimizer-ab-test)
  • instructions
  • output_modalities (["audio", "text"], ["audio"], or ["text"])
  • Audio input and output configuration — voice, TTS model, PCM format, speed
  • max_output_tokens ("inf" or a numeric ceiling)
  • tools (function definitions) and tool_choice settings
  • providerData — Inworld extensions for STT, TTS, memory, back-channel, and responsiveness (see Inworld Realtime API Extensions)
Partial updates are supported, so you can adjust the LLM, voice, TTS model, temperature, or tool lists mid-session without rebuilding the socket. 
ws.send(JSON.stringify({
  type: 'session.update',
  session: {
    type: 'realtime',
    model: 'openai/gpt-4o-mini',
    instructions: 'You are a friendly narrator.',
    output_modalities: ['audio', 'text'],
    temperature: 0.8,
    audio: {
      input: {
        transcription: { model: 'inworld/inworld-stt-1' },
        turn_detection: {
          type: 'semantic_vad',
          create_response: true,
          interrupt_response: true
        }
      },
      output: {
        voice: 'Clive',
        model: 'inworld-tts-2',
        speed: 1.0
      }
    }
  }
}));

​
STT (Speech-to-Text)

​
Choose an STT model

Set audio.input.transcription.model to select the speech-to-text model used to transcribe user audio. inworld/inworld-stt-1 is the recommended default for most realtime voice agents; pick a third-party model when its specific strength (sub-300ms latency, semantic end-of-turn, etc.) matters for your use case. 
ws.send(JSON.stringify({
  type: 'session.update',
  session: {
    audio: { input: { transcription: { model: 'inworld/inworld-stt-1' } } }
  }
}));
ModelBest for
inworld/inworld-stt-1Inworld’s first-party STT with configurable turn-taking controls. Recommended default.
assemblyai/u3-rt-proHigh-accuracy, sub-300ms latency, multilingual streaming (English, Spanish, French, German, Italian, Portuguese)
assemblyai/universal-streaming-multilingualMultilingual streaming across the same six languages
assemblyai/universal-streaming-englishEnglish-optimized streaming
soniox/stt-rt-v4High-accuracy real-time streaming with semantic end-of-turn detection and multilingual support
If the selected model is not recognised, the server responds with an error event (type: "invalid_request_error", code: "invalid_value", param: "session.audio.input.transcription.model") and the rest of the session.update is not applied. See STT Introduction for the full model catalogue and comparison.

​
Transcription hints

Guide the STT decoder with a prompt (vocabulary, domain context, formatting preferences). This is the OpenAI-standard audio.input.transcription.prompt field and is portable across OpenAI-compatible SDKs: 
ws.send(JSON.stringify({
  type: 'session.update',
  session: {
    audio: {
      input: {
        transcription: {
          model: 'assemblyai/u3-rt-pro',
          prompt: 'Medical dictation. Vocabulary: angioplasty, myocardial infarction.',
          language: 'en'
        }
      }
    }
  }
}));
FieldTypeDescription
modelstringSTT model ID. See Choose an STT model.
promptstringTranscription guidance: vocabulary hints, domain context, formatting preferences.
languagestringBCP-47 language code (e.g. "en", "es"). Optional; the model auto-detects when omitted.

​
Tune turn detection

Turn detection — when the server decides a user has finished speaking — is controlled by the OpenAI-standard audio.input.turn_detection object. The Realtime API supports both VAD types and is wire-compatible with the OpenAI SDK.

​
semantic_vad

Model-based end-of-turn detection backed by the STT stream. eagerness is the primary tuning knob. 
ws.send(JSON.stringify({
  type: 'session.update',
  session: {
    audio: {
      input: {
        turn_detection: {
          type: 'semantic_vad',
          eagerness: 'medium',          // 'low' | 'medium' | 'high' | 'auto'
          create_response: true,
          interrupt_response: true
        }
      }
    }
  }
}));
FieldTypeDescription
typestring"semantic_vad" (default)
eagernessstringHow aggressively to end turns: "low", "medium", "high", "auto". Lower eagerness requires stronger end-of-turn confidence; higher eagerness commits to end-of-turn sooner.
create_responsebooleanAuto-create a response on turn end (default true)
interrupt_responsebooleanInterrupt the active response when the user speaks (default true)
eagerness maps to a full set of four STT turn-detection parameters — confidence threshold, VAD threshold, minimum end-of-turn silence, and maximum within-turn silence. Lower thresholds and shorter silences mean the STT model commits to end-of-turn sooner (more eager).
eagernessend_of_turn_confidence_thresholdvad_thresholdmin_end_of_turn_silence (ms)max_turn_silence (ms)
low0.850.54003000
medium0.700.51602400
auto0.700.51602400
high0.550.3801200
auto mirrors medium until router-side adaptive logic exists. Any explicit field under providerData.stt (see STT extensions below) overrides the eagerness-derived default for that field — fields you do not set keep the eagerness mapping.

​
server_vad

Inworld-hosted Silero VAD + Smart Turn detector. Tunable fields match OpenAI’s server_vad shape and can be changed mid-session via partial session.update. 
ws.send(JSON.stringify({
  type: 'session.update',
  session: {
    audio: {
      input: {
        turn_detection: {
          type: 'server_vad',
          threshold: 0.5,
          prefix_padding_ms: 200,
          silence_duration_ms: 1000,
          idle_timeout_ms: 8000,
          create_response: true,
          interrupt_response: true
        }
      }
    }
  }
}));
FieldTypeDescription
typestring"server_vad"
thresholdnumberSilero VAD speech cutoff, 0.0–1.0. Default 0.5.
prefix_padding_msintegerPre-speech audio retained before an utterance, in ms. Default 200.
silence_duration_msintegerTrailing silence required to finalize the turn, in ms. Default 1000.
idle_timeout_msinteger | nullWhen set, the server emits input_audio_buffer.timeout_triggered after this many ms with no detected speech. null or 0 disables.
create_responsebooleanAuto-create a response on turn end (default true)
interrupt_responsebooleanInterrupt the active response when the user speaks (default true)
All fields accept partial session.update — omit a field to keep its current value. Changes take effect on the next audio chunk processed. See Voice Activity Detection (VAD) for the VAD event lifecycle.

​
Audio input formats

Set the wire format for client → server audio under audio.input.format. Four formats are supported; pick based on your source. The same catalog applies to audio.output.format (covered under TTS below).
typeEncodingSample rateWhen to use
audio/pcmSigned 16-bit little-endian PCMrate (default 24000)Default for browser, mobile, and most server-side sources. Send mono.
audio/pcmuG.711 μ-lawFixed 8000 Hz (ignore rate)Telephony (Twilio Media Streams, SIP trunks in North America/Japan).
audio/pcmaG.711 A-lawFixed 8000 Hz (ignore rate)Telephony (SIP trunks in Europe and most of the rest of the world).
audio/float3232-bit float PCM, little-endianrate (default 24000)Pipelines that natively produce float32 samples (some audio frameworks).
Audio is always mono, base64-encoded inside the JSON envelope (e.g. input_audio_buffer.append). 
// PCM16 @ 24 kHz (default — omit `format` entirely for the same result)
audio: { input: { format: { type: 'audio/pcm', rate: 24000 } } }

// G.711 μ-law @ 8 kHz, for Twilio
audio: { input: { format: { type: 'audio/pcmu' } } }
A legacy shorthand is also accepted: send format as a bare string — "pcm16", "g711_ulaw", "g711_alaw", or "float32" — and the server expands it to the object form above. The server resamples to 16 kHz internally for STT, so PCM input rate doesn’t need to match the STT model’s native rate. Send any rate that’s convenient; 24000 and 8000 (G.711) are the common choices. See Telephony with Twilio for a worked example of the G.711 path.

​
Send audio input

There are two ways to send audio input: Method 1: Streaming Audio (Real-time) Use input_audio_buffer.* events for streaming real-time audio from a microphone:
  1. Encode microphone data in your chosen input format (PCM16 at 24 kHz is the default).
  2. Send chunks via input_audio_buffer.append.
  3. VAD automatically detects speech boundaries and commits the buffer.
Method 2: Pre-recorded Audio Use conversation.item.create with input_audio content type for pre-recorded audio chunks: 
ws.send(JSON.stringify({
  type: 'conversation.item.create',
  item: {
    type: 'message',
    role: 'user',
    content: [{
      type: 'input_audio',
      audio: base64AudioData  // Base64-encoded PCM16 or OPUS
    }]
  }
}));

​
STT extensions

Inworld extensions for STT live under providerData.stt — voice profile signals, language hints (Soniox), and explicit overrides for the four turn-detection parameters that semantic_vad.eagerness controls implicitly. Full field reference and the voice-profile payload shape are in providerData.stt.

​
LLM

​
Choose a router or LLM

Set model in session.update to select which Router or LLM handles the conversation. The format is provider/modelName or inworld/routerId: 
ws.send(JSON.stringify({
  type: 'session.update',
  session: {
    model: 'openai/gpt-4o-mini'
  }
}));
If you omit model, the default model (google-ai-studio/gemini-2.5-flash) is used. You can change the model mid-session with a partial update — the new model takes effect on the next response.

​
Send text input

Create explicit conversation items for text turns: 
ws.send(JSON.stringify({
  type: 'conversation.item.create',
  item: {
    type: 'message',
    role: 'user',
    content: [{
      type: 'input_text',
      text: 'Give me a two-sentence summary.'
    }]
  }
}));

​
Function calling

The Realtime API supports function calling so your agent can fetch live data or trigger actions mid-conversation. Define functions in session.tools, then handle calls as they arrive.

​
1. Register a tool

ws.send(JSON.stringify({
  type: 'session.update',
  session: {
    type: 'realtime',
    tools: [{
      type: 'function',
      name: 'get_horoscope',
      description: 'Get the horoscope for a zodiac sign',
      parameters: {
        type: 'object',
        properties: {
          sign: {
            type: 'string',
            description: 'Zodiac sign, e.g. Aries, Taurus'
          }
        },
        required: ['sign']
      }
    }],
    tool_choice: 'auto'
  }
}));

​
2. Handle the function call

When the model decides to call a function, you receive a response.function_call_arguments.done event with the call_id, function name, and serialized arguments. Execute your logic, then return the result: 
ws.on('message', (buffer) => {
  const event = JSON.parse(buffer.toString());

  if (event.type === 'response.function_call_arguments.done') {
    const { call_id, name, arguments: argsJson } = event;
    const args = JSON.parse(argsJson);

    // Run your business logic
    let result;
    if (name === 'get_horoscope') {
      result = fetchHoroscope(args.sign);
    }

    // Send the function result back
    ws.send(JSON.stringify({
      type: 'conversation.item.create',
      item: {
        type: 'function_call_output',
        call_id,
        output: JSON.stringify(result)
      }
    }));

    // Tell the model to continue with the result
    ws.send(JSON.stringify({ type: 'response.create' }));
  }
});

​
3. What happens next

After response.create, the model incorporates the function output and continues the conversation — speaking the horoscope aloud (if output_modalities includes audio) or streaming text deltas. The user hears the answer without any gap in the conversation flow. You can register multiple tools and the model will call them as needed. Each call arrives as a separate response.function_call_arguments.done event with its own call_id.

​
Memory

Inworld’s automatic conversation memory layer extracts durable facts and a rolling summary, prepends them to the system prompt, and trims older transcript items so context stays bounded. Configured under providerData.memory. See providerData.memory for the field reference, and Long-term Memory for the cross-session persistence pattern.

​
TTS (Text-to-Speech)

​
Choose a TTS model

Set audio.output.model to select the text-to-speech model: 
ws.send(JSON.stringify({
  type: 'session.update',
  session: {
    audio: { output: { model: 'inworld-tts-2' } }
  }
}));
ModelSizeNotes
inworld-tts-28BHigher quality audio. Recommended for most agents. Required for providerData.tts.conversational and the CREATIVE delivery mode.
inworld-tts-1.5-mini1BFaster inference, lower latency. Server default when audio.output.model is omitted.
Examples throughout these docs use inworld-tts-2 for quality; switch to inworld-tts-1.5-mini if you’re optimizing for raw latency or running at high concurrency. You can change the TTS model mid-session alongside voice or independently.

​
Choose a voice

Set audio.output.voice to control the agent’s speaking voice: 
ws.send(JSON.stringify({
  type: 'session.update',
  session: {
    audio: { output: { voice: 'Olivia' } }
  }
}));
The default voice is Dennis. Browse available voices in the TTS Playground or list them programmatically with the List Voices API.

​
Audio output format

Set the wire format for server → client audio under audio.output.format. The catalog is identical to Audio input formats above — audio/pcm, audio/pcmu, audio/pcma, or audio/float32. Default is PCM16 at 24 kHz. 
// Default — PCM16 @ 24 kHz (omit format entirely for the same result)
audio: { output: { format: { type: 'audio/pcm', rate: 24000 } } }

// G.711 μ-law @ 8 kHz, for Twilio — TTS audio comes back already mulaw-encoded
audio: { output: { format: { type: 'audio/pcmu' } } }
In most setups, set input and output to the same format so your client only has one codec path. The exception is telephony, where you typically want both sides on G.711 to match the carrier. The server resamples internally — TTS models synthesize at their native rate and the server downsamples (or upsamples) to whatever audio.output.format.rate you request, so any reasonable rate is accepted.

​
TTS extensions

Inworld extensions for TTS live under providerData.tts — segmentation strategy, steering handling, synthesis language, the TTS-2 delivery preset, and (for TTS-2) conversational mode that preserves a shared upstream context across turns. Full field reference, segmenter strategy table, and conversational-mode details are in providerData.tts.

​
Managing the session

​
Conversation state

Use conversation events to keep context lean:
  • conversation.item.retrieve: pull any prior item by ID.
  • conversation.item.delete: remove items that should not remain in context.
Pair these with max_output_tokens and response.cancel to control overall cost (conversation management guide).

​
Observing usage

response.done carries a response.usage block on every response — including cancelled responses (barge-in, supersede). The base fields (total_tokens, input_tokens, output_tokens, plus input_token_details / output_token_details) cover LLM accounting, and three optional sub-objects attribute usage per modality:
FieldTypeDescription
usage.llm.modelstringEffective upstream LLM after router resolution. Useful when you sent inworld/auto and want to see which model the router picked.
usage.tts.modelstringTTS model used (e.g. inworld-tts-2).
usage.tts.charactersintegerCharacters synthesized across all TTS segments of this response.
usage.tts.audio_secondsnumberAssistant audio duration emitted by TTS, in seconds. The canonical TTS billing signal.
usage.stt.modelstringSTT model used (e.g. soniox/stt-rt-v4).
usage.stt.audio_secondsnumberUser audio duration transcribed for this turn, in seconds. Drained per response.done from a rolling per-session counter — each response sees only the user audio that arrived since the previous response.done.
Each modality sub-object is omitted when there’s nothing to report (e.g. a TTS-only response with no preceding user turn won’t carry stt). 
ws.on('message', (buffer) => {
  const event = JSON.parse(buffer.toString());
  if (event.type !== 'response.done') return;

  const u = event.response.usage;
  if (!u) return;
  console.log(
    `[usage] llm=${u.llm?.model ?? '?'} ` +
    `tokens=${u.input_tokens}/${u.output_tokens} ` +
    `tts=${u.tts?.audio_seconds?.toFixed(2) ?? '0'}s ` +
    `stt=${u.stt?.audio_seconds?.toFixed(2) ?? '0'}s`
  );
});
For the full schema including input_token_details / output_token_details breakdowns, see the response.done event.

​
Monitor errors

Handle error events (with type, code, and param) and implement a reconnection/backoff strategy for transient failures. See the API reference for error event schemas. 
ws.on('message', (buffer) => {
  const event = JSON.parse(buffer.toString());

  if (event.type === 'error') {
    handleError(event.error);
  }
});