Realtime TTS-2 is live. Built for realtime conversation that feels human. Read the Realtime TTS-2 announcement

WebSocket

Realtime API (WebSocket)

WSSwss://api.inworld.ai/api/v1/realtime/session

Use a persistent WebSocket connection for low-latency speech-to-speech conversations through an STT → LLM → TTS pipeline, with audio and text input and output.

Clients can:

  • Create and configure sessions with custom instructions and voice settings.
  • Stream audio input for live voice conversations, or send text input.
  • Receive streaming audio and text response events as they are generated.
  • Use automatic speech detection with configurable thresholds.
  • Control conversation flow with turn detection and response controls.
  • Keep conversation context across multiple interactions.

Rate limits: Concurrent session limits vary by subscription plan. See features and limits by plan for the table.

Inworld extensions: The session object accepts providerData for STT tuning, TTS segmentation and steering, automatic memory, back-channel responses, and responsiveness fillers. These extend the OpenAI-compatible session format. See API Extensions for each field.

This API implements the Realtime interface. See the Realtime overview for hands-on guides.

Client messages

session.update

Update the session configuration. The server responds with a session.updated event.

Payload

typeobjectrequired

event_idstring

Optional client-generated event ID.

sessionobjectrequired

The session object configures model behavior, audio settings, tools, and more. It appears in session.update, session.created, and session.updated events.

Show child attributes

objectstring

Object type identifier (read-only).

typestring

Fixed value.

idstring

Server-assigned session ID (read-only).

modelstring

Model identifier.

instructionsstring

System instructions for the model.

output_modalitiesenum<string>[]

Output types: "text", "audio", or both.

temperaturenumber

The sampling temperature used for response generation.

max_output_tokensoneOf

Maximum tokens for a response. Either an integer (1–4096) or the literal string "inf".

audioobject

Show child attributes

inputobject

Show child attributes

formatoneOf

Audio wire format for client input or server output. Used on both audio.input.format and audio.output.format. Accepts either an object form (with type + optional rate) or a legacy string alias.

noise_reductionobject

Show child attributes

typeenum<string>required

Noise reduction mode.

Available options:near_fieldfar_field

transcriptionobject

Show child attributes

modelstring

Transcription model identifier (e.g., inworld/inworld-stt-1).

languagestring

Optional language code.

promptstring

Not supported by Inworld STT. Use audio.input.transcription.prompts instead.

promptsstring[]

Expected words or phrases to help Inworld STT recognize names and domain-specific vocabulary.

turn_detectiononeOf

outputobject

Show child attributes

formatoneOf

Audio wire format for client input or server output. Used on both audio.input.format and audio.output.format. Accepts either an object form (with type + optional rate) or a legacy string alias.

voicestring

Voice preset for audio output (e.g., Dennis). See the List Voices API or the Voice library page in the Inworld Portal for the full list of supported voices.

modelstring

The TTS model used for audio output.

speednumber

Playback speed (0.25–1.5).

toolsobject[]

Show child attributes

typestringrequired

Tool type.

namestringrequired

Function name.

descriptionstring

What the function does.

parametersobject

JSON Schema for function parameters.

tool_choiceoneOf

Tool selection. Either a shorthand string (none, auto, required) or a ToolChoiceTarget object that names a specific function or MCP server.

truncationoneOf

Conversation truncation strategy. Either a shorthand string (auto, disabled) or a RetentionRatioTruncation object that pins a fraction of the context window.

tracingoneOf

Tracing configuration. Either the literal "auto" (server defaults) or a TracingConfig object with explicit workflow / group / metadata.

includeenum<string>[]

Optional data to include.

promptstringnull

Prompt template reference, or null.

providerDataobject

Inworld-specific extensions to the OpenAI-compatible session shape. Most fields are hot-swappable via partial session.update. See Inworld Realtime API Extensions for the field-by-field reference.

Show child attributes

sttobject

STT extensions: voice_profile, end_of_turn_confidence_threshold, vad_threshold, min_end_of_turn_silence, max_turn_silence. See STT provider data.

ttsobject

TTS segmentation, language, delivery, and alignment controls: segmenter_strategy, steering_handling, language, delivery_mode, conversational, user_turn_mode, timestamp_type, timestamp_transport_strategy. See TTS provider data.

memoryobject

Automatic conversation memory and summarization: enabled, turn_interval, max_memory_length, max_transcript_items, max_facts, trim_after_summarize. See Memory provider data.

backchannelobject

Opt-in back-channel responses (brief acknowledgements while the user is speaking). See Back-channel.

responsivenessobject

Opt-in responsiveness fillers (low-latency acknowledgements emitted before the main response). See Responsiveness.

auto_tool_responsebooleandefault: true

When true, adding a functioncalloutput automatically starts the follow-up response. Set false for OpenAI-compatible client-controlled continuation, then send response.create explicitly. Omitted partial updates preserve the current value.

text_generation_configobject

Fine-grained LLM generation parameters. Also accepted under providerData.text_generation_config.

Show child attributes

reasoningobject

Chain-of-thought reasoning control. Forwarded to the LLM Router as extra_body.reasoning.

Show child attributes

effortenum<string>

Reasoning depth. Higher values allocate more thinking tokens.

Available options:NONEMINIMALLOWMEDIUMHIGHXHIGH

maxTokensinteger

Cap on reasoning/thinking tokens.

excludeboolean

When true, reasoning tokens are generated but excluded from response text.

temperaturenumber

Sampling temperature override.

topPnumber

Nucleus sampling.

maxNewTokensinteger

Max completion tokens.

frequencyPenaltynumber

Frequency penalty.

presencePenaltynumber

Presence penalty.

expires_atinteger

Unix timestamp for session expiration (read-only).

Example

{
  "type": "session.update",
  "session": {
    "instructions": "You are a friendly voice assistant.",
    "audio": {
      "input": {
        "transcription": {
          "model": "inworld/inworld-stt-1"
        },
        "turn_detection": {
          "type": "semantic_vad",
          "eagerness": "medium",
          "create_response": true,
          "interrupt_response": true
        }
      },
      "output": {
        "model": "inworld-tts-2",
        "voice": "Dennis",
        "speed": 1
      }
    }
  }
}
conversation.item.create

Add a conversation item (message, tool call, or tool result).

Payload

typeobjectrequired

event_idstring

Optional client-generated event ID.

previous_item_idstring

Controls insertion. Omit to append, use "root" to prepend, or provide an existing item ID to insert immediately after it. An unknown ID returns itemnotfound with error.param set to previousitemid and does not mutate history.

itemobjectrequired

Client-created conversation item. functioncall requires a non-empty name; functioncalloutput requires a non-empty callid. arguments and output may be omitted and normalize to empty strings.

Show child attributes

objectstring

Object type identifier (read-only, present in server responses).

idstring

Item ID.

typeenum<string>required

Item variant.

Available options:messagefunction_callfunction_call_output

statusenum<string>

Item status (read-only, present in server responses).

Available options:completedin_progressincomplete

roleenum<string>

Available options:systemuserassistanttool

contentobject[]

Show child attributes

typeenum<string>required

Content type. User input uses inputtext/inputaudio/inputimage. Assistant item snapshots use outputtext/outputaudio. Streaming response.contentpart events use text/audio. Legacy text/audio aliases are accepted when creating assistant items.

Available options:input_textinput_audioinput_imagetextaudiooutput_textoutput_audio

textstring

Text content.

audiostring

Base64-encoded audio.

transcriptstring

Human-readable transcript accompanying audio.

imagestring

Inline image data for input_image content.

image_urlstring

Image URL for input_image content.

detailenum<string>

Image detail level.

Available options:lowhighauto

call_idstring

Tool-call identifier. Required and non-empty for client-created functioncalloutput; optional for functioncall. Always present on server functioncall_output snapshots, including as an empty string.

namestring

Function name. Required and non-empty for client-created functioncall; always present on server functioncall snapshots, including as an empty string.

argumentsstring

Function arguments as a JSON string. May be omitted on input and then normalizes to an empty string; always present on server function_call snapshots.

outputstring

Function result. May be omitted on input and then normalizes to an empty string; always present on server functioncalloutput snapshots.

force_uninterruptibleboolean

Inworld extension (read-only). Present and true on the assistant message produced by a response.speak request that set force_uninterruptible; omitted otherwise. It describes that response only and does not protect later responses.

Example

{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "content": [
      {
        "type": "input_text",
        "text": "Hello, how are you?"
      }
    ]
  }
}
conversation.item.truncate

Truncate an assistant message's audio.

Payload

typeobjectrequired

event_idstring

Optional client-generated event ID.

item_idstringrequired

The ID of the assistant message item to truncate.

content_indexintegerrequired

Index of the content part to truncate.

audio_end_msintegerrequired

Millisecond offset to truncate the audio at.

Example

{
  "event_id": "string",
  "item_id": "string",
  "content_index": 0,
  "audio_end_ms": 0
}
conversation.item.delete

Delete a conversation item by ID.

Payload

typeobjectrequired

event_idstring

Optional client-generated event ID.

item_idstringrequired

The ID of the conversation item to delete.

Example

{
  "event_id": "string",
  "item_id": "string"
}
conversation.item.retrieve

Retrieve a conversation item by ID.

Payload

typeobjectrequired

event_idstring

Optional client-generated event ID.

item_idstringrequired

The ID of the conversation item to retrieve.

Example

{
  "event_id": "string",
  "item_id": "string"
}
response.create

Trigger a model response. The server streams back response events.

Payload

typeobjectrequired

event_idstring

Optional client-generated event ID.

responseobject

Per-response overrides for session defaults.

Show child attributes

conversationoneOf

Conversation selector. Either the literal "auto" (use the session's implicit conversation) or an explicit conversation ID string.

output_modalitiesenum<string>[]

"text", "audio", or both.

instructionsstring

Override instructions for this response.

voicestring

Override voice for this response.

max_output_tokensoneOf

Maximum tokens for a response. Either an integer (1–4096) or the literal string "inf".

tool_choiceoneOf

Tool selection. Either a shorthand string (none, auto, required) or a ToolChoiceTarget object that names a specific function or MCP server.

toolsobject[]

Override available tools.

Show child attributes

typestringrequired

Tool type.

namestringrequired

Function name.

descriptionstring

What the function does.

parametersobject

JSON Schema for function parameters.

metadataobject

Client correlation metadata echoed on response.created and response.done. Keys and values are strings; up to 16 entries, 64 Unicode code points per key, and 512 per value.

Example

{
  "type": "response.create",
  "response": {
    "output_modalities": [
      "audio",
      "text"
    ],
    "instructions": "Respond in a cheerful tone."
  }
}
response.cancel

Cancel an in-progress response.

Payload

typeobjectrequired

event_idstring

Optional client-generated event ID.

response_idstring

Cancel a specific response by ID. If omitted, cancels the active response.

Example

{
  "event_id": "string",
  "response_id": "string"
}
response.speak

Inworld extension. Speak the supplied text verbatim as the next assistant response, without calling the LLM. The server acknowledges the request with response.speak.acknowledged, then streams the normal response events. Audio sessions synthesize the text with the session's TTS configuration; text-only sessions receive it as a single text delta.

Payload

typeobjectrequired

event_idstring

Optional client-generated event ID. Echoed as request_event_id on response.speak.acknowledged. It is not an idempotency key: repeated requests are spoken again.

textstringrequired

Text to speak, verbatim. Sent to TTS unchanged, with no LLM rewriting. Required and must be a string; an empty string is accepted. There is no feature-specific length limit. A non-string value fails decoding: the server returns an error event with code invalid_json instead of an acknowledgment.

force_uninterruptiblebooleandefault: false

When true, speech detection and new user messages do not cancel this response, and automatically triggered responses (turn detection, tool continuation) are rejected with conversation_already_has_active_response while it is generating. Explicit response.cancel, response.create, and response.speak still cancel or replace it. Protection ends when generation ends, not when the client finishes playback. When false or omitted, the response is interrupted like any other.

metadataobject

Client correlation metadata echoed on response.speak.acknowledged, response.created, and response.done. Same limits as response.metadata on response.create: up to 16 entries, 64 Unicode code points per key, and 512 per value.

Example

{
  "type": "response.speak",
  "event_id": "evt_greeting_1",
  "text": "Hi! Thanks for calling. How can I help you today?",
  "force_uninterruptible": true,
  "metadata": {
    "purpose": "greeting"
  }
}
input_audio_buffer.append

Append audio bytes to the input buffer.

Payload

typeobjectrequired

event_idstring

Optional client-generated event ID.

audiostringrequired

Base64-encoded audio chunk (~100–200ms) matching the configured input format.

Example

{
  "event_id": "string",
  "audio": "string"
}
input_audio_buffer.commit

Commit the buffered audio as a user message.

Payload

typeobjectrequired

event_idstring

Example

{
  "event_id": "string"
}
input_audio_buffer.clear

Discard all audio in the input buffer.

Payload

typeobjectrequired

event_idstring

Example

{
  "event_id": "string"
}
output_audio_buffer.clear

Clear the server's output audio buffer, stopping playback.

Payload

typeobjectrequired

event_idstring

Example

{
  "event_id": "string"
}

Server messages

session.created

Sent by the server immediately when the WebSocket connection is established, carrying the session's default configuration. Send a session.update to configure the session.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

sessionobjectrequired

The session object configures model behavior, audio settings, tools, and more. It appears in session.update, session.created, and session.updated events.

Show child attributes

objectstring

Object type identifier (read-only).

typestring

Fixed value.

idstring

Server-assigned session ID (read-only).

modelstring

Model identifier.

instructionsstring

System instructions for the model.

output_modalitiesenum<string>[]

Output types: "text", "audio", or both.

temperaturenumber

The sampling temperature used for response generation.

max_output_tokensoneOf

Maximum tokens for a response. Either an integer (1–4096) or the literal string "inf".

audioobject

Show child attributes

inputobject

Show child attributes

formatoneOf

Audio wire format for client input or server output. Used on both audio.input.format and audio.output.format. Accepts either an object form (with type + optional rate) or a legacy string alias.

noise_reductionobject

Show child attributes

typeenum<string>required

Noise reduction mode.

Available options:near_fieldfar_field

transcriptionobject

Show child attributes

modelstring

Transcription model identifier (e.g., inworld/inworld-stt-1).

languagestring

Optional language code.

promptstring

Not supported by Inworld STT. Use audio.input.transcription.prompts instead.

promptsstring[]

Expected words or phrases to help Inworld STT recognize names and domain-specific vocabulary.

turn_detectiononeOf

outputobject

Show child attributes

formatoneOf

Audio wire format for client input or server output. Used on both audio.input.format and audio.output.format. Accepts either an object form (with type + optional rate) or a legacy string alias.

voicestring

Voice preset for audio output (e.g., Dennis). See the List Voices API or the Voice library page in the Inworld Portal for the full list of supported voices.

modelstring

The TTS model used for audio output.

speednumber

Playback speed (0.25–1.5).

toolsobject[]

Show child attributes

typestringrequired

Tool type.

namestringrequired

Function name.

descriptionstring

What the function does.

parametersobject

JSON Schema for function parameters.

tool_choiceoneOf

Tool selection. Either a shorthand string (none, auto, required) or a ToolChoiceTarget object that names a specific function or MCP server.

truncationoneOf

Conversation truncation strategy. Either a shorthand string (auto, disabled) or a RetentionRatioTruncation object that pins a fraction of the context window.

tracingoneOf

Tracing configuration. Either the literal "auto" (server defaults) or a TracingConfig object with explicit workflow / group / metadata.

includeenum<string>[]

Optional data to include.

promptstringnull

Prompt template reference, or null.

providerDataobject

Inworld-specific extensions to the OpenAI-compatible session shape. Most fields are hot-swappable via partial session.update. See Inworld Realtime API Extensions for the field-by-field reference.

Show child attributes

sttobject

STT extensions: voice_profile, end_of_turn_confidence_threshold, vad_threshold, min_end_of_turn_silence, max_turn_silence. See STT provider data.

ttsobject

TTS segmentation, language, delivery, and alignment controls: segmenter_strategy, steering_handling, language, delivery_mode, conversational, user_turn_mode, timestamp_type, timestamp_transport_strategy. See TTS provider data.

memoryobject

Automatic conversation memory and summarization: enabled, turn_interval, max_memory_length, max_transcript_items, max_facts, trim_after_summarize. See Memory provider data.

backchannelobject

Opt-in back-channel responses (brief acknowledgements while the user is speaking). See Back-channel.

responsivenessobject

Opt-in responsiveness fillers (low-latency acknowledgements emitted before the main response). See Responsiveness.

auto_tool_responsebooleandefault: true

When true, adding a functioncalloutput automatically starts the follow-up response. Set false for OpenAI-compatible client-controlled continuation, then send response.create explicitly. Omitted partial updates preserve the current value.

text_generation_configobject

Fine-grained LLM generation parameters. Also accepted under providerData.text_generation_config.

Show child attributes

reasoningobject

Chain-of-thought reasoning control. Forwarded to the LLM Router as extra_body.reasoning.

Show child attributes

effortenum<string>

Reasoning depth. Higher values allocate more thinking tokens.

Available options:NONEMINIMALLOWMEDIUMHIGHXHIGH

maxTokensinteger

Cap on reasoning/thinking tokens.

excludeboolean

When true, reasoning tokens are generated but excluded from response text.

temperaturenumber

Sampling temperature override.

topPnumber

Nucleus sampling.

maxNewTokensinteger

Max completion tokens.

frequencyPenaltynumber

Frequency penalty.

presencePenaltynumber

Presence penalty.

expires_atinteger

Unix timestamp for session expiration (read-only).

Example

{
  "event_id": "string",
  "session": {
    "object": "string",
    "type": "string",
    "id": "string",
    "model": "string",
    "instructions": "string",
    "output_modalities": [
      "text"
    ],
    "temperature": 0,
    "max_output_tokens": 0,
    "audio": {
      "input": {
        "format": {
          "type": "audio/pcm",
          "rate": 0
        },
        "noise_reduction": {
          "type": "near_field"
        },
        "transcription": {
          "model": "string",
          "language": "string",
          "prompt": "string",
          "prompts": [
            "string"
          ]
        },
        "turn_detection": {
          "threshold": 0,
          "prefix_padding_ms": 0,
          "silence_duration_ms": 0,
          "create_response": true,
          "interrupt_response": true,
          "idle_timeout_ms": 0
        }
      },
      "output": {
        "format": {
          "type": "audio/pcm",
          "rate": 0
        },
        "voice": "string",
        "model": "string",
        "speed": 0
      }
    },
    "tools": [
      {
        "type": "string",
        "name": "string",
        "description": "string"
      }
    ],
    "tool_choice": "none",
    "truncation": "auto",
    "tracing": "string",
    "include": [
      "item.input_audio_transcription.logprobs"
    ],
    "providerData": {
      "auto_tool_response": true
    },
    "text_generation_config": {
      "reasoning": {
        "effort": "NONE",
        "maxTokens": 0,
        "exclude": true
      },
      "temperature": 0,
      "topP": 0,
      "maxNewTokens": 0,
      "frequencyPenalty": 0,
      "presencePenalty": 0
    },
    "expires_at": 0
  }
}
session.updated

Confirms a session.update was applied.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

sessionobjectrequired

The session object configures model behavior, audio settings, tools, and more. It appears in session.update, session.created, and session.updated events.

Show child attributes

objectstring

Object type identifier (read-only).

typestring

Fixed value.

idstring

Server-assigned session ID (read-only).

modelstring

Model identifier.

instructionsstring

System instructions for the model.

output_modalitiesenum<string>[]

Output types: "text", "audio", or both.

temperaturenumber

The sampling temperature used for response generation.

max_output_tokensoneOf

Maximum tokens for a response. Either an integer (1–4096) or the literal string "inf".

audioobject

Show child attributes

inputobject

Show child attributes

formatoneOf

Audio wire format for client input or server output. Used on both audio.input.format and audio.output.format. Accepts either an object form (with type + optional rate) or a legacy string alias.

noise_reductionobject

Show child attributes

typeenum<string>required

Noise reduction mode.

Available options:near_fieldfar_field

transcriptionobject

Show child attributes

modelstring

Transcription model identifier (e.g., inworld/inworld-stt-1).

languagestring

Optional language code.

promptstring

Not supported by Inworld STT. Use audio.input.transcription.prompts instead.

promptsstring[]

Expected words or phrases to help Inworld STT recognize names and domain-specific vocabulary.

turn_detectiononeOf

outputobject

Show child attributes

formatoneOf

Audio wire format for client input or server output. Used on both audio.input.format and audio.output.format. Accepts either an object form (with type + optional rate) or a legacy string alias.

voicestring

Voice preset for audio output (e.g., Dennis). See the List Voices API or the Voice library page in the Inworld Portal for the full list of supported voices.

modelstring

The TTS model used for audio output.

speednumber

Playback speed (0.25–1.5).

toolsobject[]

Show child attributes

typestringrequired

Tool type.

namestringrequired

Function name.

descriptionstring

What the function does.

parametersobject

JSON Schema for function parameters.

tool_choiceoneOf

Tool selection. Either a shorthand string (none, auto, required) or a ToolChoiceTarget object that names a specific function or MCP server.

truncationoneOf

Conversation truncation strategy. Either a shorthand string (auto, disabled) or a RetentionRatioTruncation object that pins a fraction of the context window.

tracingoneOf

Tracing configuration. Either the literal "auto" (server defaults) or a TracingConfig object with explicit workflow / group / metadata.

includeenum<string>[]

Optional data to include.

promptstringnull

Prompt template reference, or null.

providerDataobject

Inworld-specific extensions to the OpenAI-compatible session shape. Most fields are hot-swappable via partial session.update. See Inworld Realtime API Extensions for the field-by-field reference.

Show child attributes

sttobject

STT extensions: voice_profile, end_of_turn_confidence_threshold, vad_threshold, min_end_of_turn_silence, max_turn_silence. See STT provider data.

ttsobject

TTS segmentation, language, delivery, and alignment controls: segmenter_strategy, steering_handling, language, delivery_mode, conversational, user_turn_mode, timestamp_type, timestamp_transport_strategy. See TTS provider data.

memoryobject

Automatic conversation memory and summarization: enabled, turn_interval, max_memory_length, max_transcript_items, max_facts, trim_after_summarize. See Memory provider data.

backchannelobject

Opt-in back-channel responses (brief acknowledgements while the user is speaking). See Back-channel.

responsivenessobject

Opt-in responsiveness fillers (low-latency acknowledgements emitted before the main response). See Responsiveness.

auto_tool_responsebooleandefault: true

When true, adding a functioncalloutput automatically starts the follow-up response. Set false for OpenAI-compatible client-controlled continuation, then send response.create explicitly. Omitted partial updates preserve the current value.

text_generation_configobject

Fine-grained LLM generation parameters. Also accepted under providerData.text_generation_config.

Show child attributes

reasoningobject

Chain-of-thought reasoning control. Forwarded to the LLM Router as extra_body.reasoning.

Show child attributes

effortenum<string>

Reasoning depth. Higher values allocate more thinking tokens.

Available options:NONEMINIMALLOWMEDIUMHIGHXHIGH

maxTokensinteger

Cap on reasoning/thinking tokens.

excludeboolean

When true, reasoning tokens are generated but excluded from response text.

temperaturenumber

Sampling temperature override.

topPnumber

Nucleus sampling.

maxNewTokensinteger

Max completion tokens.

frequencyPenaltynumber

Frequency penalty.

presencePenaltynumber

Presence penalty.

expires_atinteger

Unix timestamp for session expiration (read-only).

Example

{
  "event_id": "string",
  "session": {
    "object": "string",
    "type": "string",
    "id": "string",
    "model": "string",
    "instructions": "string",
    "output_modalities": [
      "text"
    ],
    "temperature": 0,
    "max_output_tokens": 0,
    "audio": {
      "input": {
        "format": {
          "type": "audio/pcm",
          "rate": 0
        },
        "noise_reduction": {
          "type": "near_field"
        },
        "transcription": {
          "model": "string",
          "language": "string",
          "prompt": "string",
          "prompts": [
            "string"
          ]
        },
        "turn_detection": {
          "threshold": 0,
          "prefix_padding_ms": 0,
          "silence_duration_ms": 0,
          "create_response": true,
          "interrupt_response": true,
          "idle_timeout_ms": 0
        }
      },
      "output": {
        "format": {
          "type": "audio/pcm",
          "rate": 0
        },
        "voice": "string",
        "model": "string",
        "speed": 0
      }
    },
    "tools": [
      {
        "type": "string",
        "name": "string",
        "description": "string"
      }
    ],
    "tool_choice": "none",
    "truncation": "auto",
    "tracing": "string",
    "include": [
      "item.input_audio_transcription.logprobs"
    ],
    "providerData": {
      "auto_tool_response": true
    },
    "text_generation_config": {
      "reasoning": {
        "effort": "NONE",
        "maxTokens": 0,
        "exclude": true
      },
      "temperature": 0,
      "topP": 0,
      "maxNewTokens": 0,
      "frequencyPenalty": 0,
      "presencePenalty": 0
    },
    "expires_at": 0
  }
}
error

Indicates an error occurred.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

errorobjectrequired

Show child attributes

typestringrequired

Error category.

codestring

Error code.

messagestringrequired

Human-readable error description.

paramstring

Related parameter, if applicable.

event_idstring

The client event ID that caused the error, if applicable.

Example

{
  "event_id": "2c23cfd4-a4b5-4a96-83b8-a6a151f3989e",
  "type": "error",
  "error": {
    "type": "server_error",
    "code": null,
    "message": "Failed to read content stream.",
    "param": null,
    "event_id": null
  }
}
conversation.item.added

A new item was added to the conversation.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

previous_item_idstringnull

The ID of the preceding conversation item, or null.

itemobjectrequired

Server conversation item snapshot. Variant-specific function fields are always serialized, including as empty strings.

Show child attributes

objectstring

Object type identifier (read-only, present in server responses).

idstring

Item ID.

typeenum<string>required

Item variant.

Available options:messagefunction_callfunction_call_output

statusenum<string>

Item status (read-only, present in server responses).

Available options:completedin_progressincomplete

roleenum<string>

Available options:systemuserassistanttool

contentobject[]

Show child attributes

typeenum<string>required

Content type. User input uses inputtext/inputaudio/inputimage. Assistant item snapshots use outputtext/outputaudio. Streaming response.contentpart events use text/audio. Legacy text/audio aliases are accepted when creating assistant items.

Available options:input_textinput_audioinput_imagetextaudiooutput_textoutput_audio

textstring

Text content.

audiostring

Base64-encoded audio.

transcriptstring

Human-readable transcript accompanying audio.

imagestring

Inline image data for input_image content.

image_urlstring

Image URL for input_image content.

detailenum<string>

Image detail level.

Available options:lowhighauto

call_idstring

Tool-call identifier. Required and non-empty for client-created functioncalloutput; optional for functioncall. Always present on server functioncall_output snapshots, including as an empty string.

namestring

Function name. Required and non-empty for client-created functioncall; always present on server functioncall snapshots, including as an empty string.

argumentsstring

Function arguments as a JSON string. May be omitted on input and then normalizes to an empty string; always present on server function_call snapshots.

outputstring

Function result. May be omitted on input and then normalizes to an empty string; always present on server functioncalloutput snapshots.

force_uninterruptibleboolean

Inworld extension (read-only). Present and true on the assistant message produced by a response.speak request that set force_uninterruptible; omitted otherwise. It describes that response only and does not protect later responses.

Example

{
  "event_id": "string",
  "item": {
    "object": "string",
    "id": "string",
    "type": "message",
    "status": "completed",
    "role": "system",
    "content": [
      {
        "type": "input_text",
        "text": "string",
        "audio": "string",
        "transcript": "string",
        "image": "string",
        "image_url": "string",
        "detail": "low"
      }
    ],
    "call_id": "string",
    "name": "string",
    "arguments": "string",
    "output": "string",
    "force_uninterruptible": true
  }
}
conversation.item.done

An item finished being populated.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

previous_item_idstringnull

The ID of the preceding conversation item, or null.

itemobjectrequired

Server conversation item snapshot. Variant-specific function fields are always serialized, including as empty strings.

Show child attributes

objectstring

Object type identifier (read-only, present in server responses).

idstring

Item ID.

typeenum<string>required

Item variant.

Available options:messagefunction_callfunction_call_output

statusenum<string>

Item status (read-only, present in server responses).

Available options:completedin_progressincomplete

roleenum<string>

Available options:systemuserassistanttool

contentobject[]

Show child attributes

typeenum<string>required

Content type. User input uses inputtext/inputaudio/inputimage. Assistant item snapshots use outputtext/outputaudio. Streaming response.contentpart events use text/audio. Legacy text/audio aliases are accepted when creating assistant items.

Available options:input_textinput_audioinput_imagetextaudiooutput_textoutput_audio

textstring

Text content.

audiostring

Base64-encoded audio.

transcriptstring

Human-readable transcript accompanying audio.

imagestring

Inline image data for input_image content.

image_urlstring

Image URL for input_image content.

detailenum<string>

Image detail level.

Available options:lowhighauto

call_idstring

Tool-call identifier. Required and non-empty for client-created functioncalloutput; optional for functioncall. Always present on server functioncall_output snapshots, including as an empty string.

namestring

Function name. Required and non-empty for client-created functioncall; always present on server functioncall snapshots, including as an empty string.

argumentsstring

Function arguments as a JSON string. May be omitted on input and then normalizes to an empty string; always present on server function_call snapshots.

outputstring

Function result. May be omitted on input and then normalizes to an empty string; always present on server functioncalloutput snapshots.

force_uninterruptibleboolean

Inworld extension (read-only). Present and true on the assistant message produced by a response.speak request that set force_uninterruptible; omitted otherwise. It describes that response only and does not protect later responses.

Example

{
  "event_id": "string",
  "item": {
    "object": "string",
    "id": "string",
    "type": "message",
    "status": "completed",
    "role": "system",
    "content": [
      {
        "type": "input_text",
        "text": "string",
        "audio": "string",
        "transcript": "string",
        "image": "string",
        "image_url": "string",
        "detail": "low"
      }
    ],
    "call_id": "string",
    "name": "string",
    "arguments": "string",
    "output": "string",
    "force_uninterruptible": true
  }
}
conversation.item.deleted

An item was deleted from the conversation.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

item_idstringrequired

The ID of the deleted item.

Example

{
  "event_id": "string",
  "item_id": "string"
}
conversation.item.retrieved

Response to conversation.item.retrieve.

Payload

typeobjectrequired

event_idstringrequired

Example

{
  "event_id": "string"
}
conversation.item.truncated

An assistant audio item was truncated.

Payload

typeobjectrequired

event_idstringrequired

Example

{
  "event_id": "string"
}
conversation.item.input_audio_transcription.delta

Streaming partial transcription for user audio.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

item_idstringrequired

The conversation item being transcribed.

content_indexinteger

Index of the content part being transcribed.

deltastringrequired

Partial transcription text.

Example

{
  "event_id": "string",
  "item_id": "string",
  "content_index": 0,
  "delta": "string"
}
conversation.item.input_audio_transcription.completed

Final transcription for a user audio item.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

item_idstringrequired

The conversation item that was transcribed.

content_indexinteger

Index of the content part that was transcribed.

transcriptstringrequired

Complete transcription text.

Example

{
  "event_id": "string",
  "item_id": "string",
  "content_index": 0,
  "transcript": "string"
}
response.speak.acknowledged

Inworld extension. Reports whether a response.speak request was accepted. Sent once per decoded request, before response.created for that response. Acceptance means the request was admitted, not that synthesis or playback succeeded; later failures arrive through the normal error and response.done events.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

request_event_idstring

The event_id of the response.speak request. Omitted when the request had no event_id.

acceptedbooleanrequired

true when the request was admitted and a response was started; false when it was rejected. A rejected request creates no response or conversation item and does not cancel the active response.

response_idstring

ID of the response started for this request. Present only when accepted is true.

metadataobject

The request's metadata, echoed when it is valid.

errorobject

Why the request was rejected. Present only when accepted is false.

Show child attributes

typestringrequired

Error category, e.g. invalid_request_error.

codestring

Error code, e.g. missing_required_parameter or invalid_value.

messagestringrequired

Human-readable error description.

paramstringnull

Related parameter (text or metadata), or null when the rejection is not about a field.

Example

{
  "type": "response.speak.acknowledged",
  "event_id": "evt_server_1",
  "request_event_id": "evt_greeting_1",
  "accepted": true,
  "response_id": "resp_1",
  "metadata": {
    "purpose": "greeting"
  }
}
response.created

A new response was created. Contains the full response object in its initial state.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

responseobjectrequired

The response object returned in response.created and response.done events.

Show child attributes

idstring

Response identifier.

objectstring

Object type.

statusenum<string>

Response status.

Available options:in_progresscompletedcancelledfailed

status_detailsoneOf

Status detail object for the response, or null if no details are available.

outputobject[]

Output items.

Show child attributes

objectstring

Object type identifier (read-only, present in server responses).

idstring

Item ID.

typeenum<string>required

Item variant.

Available options:messagefunction_callfunction_call_output

statusenum<string>

Item status (read-only, present in server responses).

Available options:completedin_progressincomplete

roleenum<string>

Available options:systemuserassistanttool

contentobject[]

Show child attributes

typeenum<string>required

Content type. User input uses inputtext/inputaudio/inputimage. Assistant item snapshots use outputtext/outputaudio. Streaming response.contentpart events use text/audio. Legacy text/audio aliases are accepted when creating assistant items.

Available options:input_textinput_audioinput_imagetextaudiooutput_textoutput_audio

textstring

Text content.

audiostring

Base64-encoded audio.

transcriptstring

Human-readable transcript accompanying audio.

imagestring

Inline image data for input_image content.

image_urlstring

Image URL for input_image content.

detailenum<string>

Image detail level.

Available options:lowhighauto

call_idstring

Tool-call identifier. Required and non-empty for client-created functioncalloutput; optional for functioncall. Always present on server functioncall_output snapshots, including as an empty string.

namestring

Function name. Required and non-empty for client-created functioncall; always present on server functioncall snapshots, including as an empty string.

argumentsstring

Function arguments as a JSON string. May be omitted on input and then normalizes to an empty string; always present on server function_call snapshots.

outputstring

Function result. May be omitted on input and then normalizes to an empty string; always present on server functioncalloutput snapshots.

force_uninterruptibleboolean

Inworld extension (read-only). Present and true on the assistant message produced by a response.speak request that set force_uninterruptible; omitted otherwise. It describes that response only and does not protect later responses.

conversation_idstring

Conversation this response belongs to.

output_modalitiesstring[]

"text", "audio", or both.

max_output_tokensoneOf

Maximum tokens for a response. Either an integer (1–4096) or the literal string "inf".

audioobject

Audio output config echoed from session.

usageobject

Token + audio usage for the response. The base block (total_tokens, input_tokens, output_tokens, input_token_details, output_token_details) covers LLM token accounting. The optional llm, tts, and stt sub-objects attribute usage per modality to the upstream model/provider and add tts.audio_seconds/stt.audio_seconds for audio billing. Each modality sub-object is omitted when its data is unavailable (e.g. a response with no TTS output, or a TTS-only response with no STT turn).

Show child attributes

total_tokensinteger

input_tokensinteger

output_tokensinteger

input_token_detailsobject

Show child attributes

cached_tokensinteger

Input tokens served from a prompt cache hit. Populated by both implicit provider caching and explicit caching via providerData.caching.

cache_write_tokensinteger

Input tokens written to the cache when establishing a new entry (explicit caching on supporting providers, e.g. Anthropic). Omitted on cache hits.

text_tokensinteger

audio_tokensinteger

Reserved for multimodal upstreams. Currently always 0 on Inworld's router-fronted models.

output_token_detailsobject

Show child attributes

text_tokensinteger

audio_tokensinteger

Reserved for multimodal upstreams. Currently always 0; assistant audio bytes are reported in tts.audio_seconds.

reasoning_tokensinteger

Reasoning/thinking tokens emitted by reasoning-capable upstreams.

llmobject

Attribution for the LLM portion of this response.

Show child attributes

modelstring

Effective upstream model ID after router resolution (e.g. openai/gpt-4o-mini).

ttsobject

TTS usage accumulated across all segments of this response.

Show child attributes

modelstring

TTS model used for this response (e.g. inworld-tts-2).

charactersinteger

Total characters synthesized.

audio_secondsnumber

Total assistant audio duration emitted by TTS, in seconds.

sttobject

STT usage for the user audio that fed this response. Drained per response.done from a rolling per-session counter, so each response sees only the audio transcribed since the previous response.done.

Show child attributes

modelstring

STT model used (e.g. inworld/inworld-stt-1).

audio_secondsnumber

User audio duration transcribed for this turn, in seconds.

metadataobject

Metadata echoed from response.create. Omitted when the client did not supply response metadata; never contains LLM Router providerData.metadata.

Example

{
  "event_id": "string",
  "response": {
    "id": "string",
    "object": "string",
    "status": "in_progress",
    "status_details": {
      "type": "completed",
      "reason": "string",
      "error": {
        "type": "string",
        "code": "string"
      }
    },
    "output": [
      {
        "object": "string",
        "id": "string",
        "type": "message",
        "status": "completed",
        "role": "system",
        "content": [
          {
            "type": "input_text",
            "text": "string",
            "audio": "string",
            "transcript": "string",
            "image": "string",
            "image_url": "string",
            "detail": "low"
          }
        ],
        "call_id": "string",
        "name": "string",
        "arguments": "string",
        "output": "string",
        "force_uninterruptible": true
      }
    ],
    "conversation_id": "string",
    "output_modalities": [
      "string"
    ],
    "max_output_tokens": 0,
    "usage": {
      "total_tokens": 0,
      "input_tokens": 0,
      "output_tokens": 0,
      "input_token_details": {
        "cached_tokens": 0,
        "cache_write_tokens": 0,
        "text_tokens": 0,
        "audio_tokens": 0
      },
      "output_token_details": {
        "text_tokens": 0,
        "audio_tokens": 0,
        "reasoning_tokens": 0
      },
      "llm": {
        "model": "string"
      },
      "tts": {
        "model": "string",
        "characters": 0,
        "audio_seconds": 0
      },
      "stt": {
        "model": "string",
        "audio_seconds": 0
      }
    }
  }
}
response.done

The response finished. Contains the completed response object with final status and output.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

responseobjectrequired

The response object returned in response.created and response.done events.

Show child attributes

idstring

Response identifier.

objectstring

Object type.

statusenum<string>

Response status.

Available options:in_progresscompletedcancelledfailed

status_detailsoneOf

Status detail object for the response, or null if no details are available.

outputobject[]

Output items.

Show child attributes

objectstring

Object type identifier (read-only, present in server responses).

idstring

Item ID.

typeenum<string>required

Item variant.

Available options:messagefunction_callfunction_call_output

statusenum<string>

Item status (read-only, present in server responses).

Available options:completedin_progressincomplete

roleenum<string>

Available options:systemuserassistanttool

contentobject[]

Show child attributes

typeenum<string>required

Content type. User input uses inputtext/inputaudio/inputimage. Assistant item snapshots use outputtext/outputaudio. Streaming response.contentpart events use text/audio. Legacy text/audio aliases are accepted when creating assistant items.

Available options:input_textinput_audioinput_imagetextaudiooutput_textoutput_audio

textstring

Text content.

audiostring

Base64-encoded audio.

transcriptstring

Human-readable transcript accompanying audio.

imagestring

Inline image data for input_image content.

image_urlstring

Image URL for input_image content.

detailenum<string>

Image detail level.

Available options:lowhighauto

call_idstring

Tool-call identifier. Required and non-empty for client-created functioncalloutput; optional for functioncall. Always present on server functioncall_output snapshots, including as an empty string.

namestring

Function name. Required and non-empty for client-created functioncall; always present on server functioncall snapshots, including as an empty string.

argumentsstring

Function arguments as a JSON string. May be omitted on input and then normalizes to an empty string; always present on server function_call snapshots.

outputstring

Function result. May be omitted on input and then normalizes to an empty string; always present on server functioncalloutput snapshots.

force_uninterruptibleboolean

Inworld extension (read-only). Present and true on the assistant message produced by a response.speak request that set force_uninterruptible; omitted otherwise. It describes that response only and does not protect later responses.

conversation_idstring

Conversation this response belongs to.

output_modalitiesstring[]

"text", "audio", or both.

max_output_tokensoneOf

Maximum tokens for a response. Either an integer (1–4096) or the literal string "inf".

audioobject

Audio output config echoed from session.

usageobject

Token + audio usage for the response. The base block (total_tokens, input_tokens, output_tokens, input_token_details, output_token_details) covers LLM token accounting. The optional llm, tts, and stt sub-objects attribute usage per modality to the upstream model/provider and add tts.audio_seconds/stt.audio_seconds for audio billing. Each modality sub-object is omitted when its data is unavailable (e.g. a response with no TTS output, or a TTS-only response with no STT turn).

Show child attributes

total_tokensinteger

input_tokensinteger

output_tokensinteger

input_token_detailsobject

Show child attributes

cached_tokensinteger

Input tokens served from a prompt cache hit. Populated by both implicit provider caching and explicit caching via providerData.caching.

cache_write_tokensinteger

Input tokens written to the cache when establishing a new entry (explicit caching on supporting providers, e.g. Anthropic). Omitted on cache hits.

text_tokensinteger

audio_tokensinteger

Reserved for multimodal upstreams. Currently always 0 on Inworld's router-fronted models.

output_token_detailsobject

Show child attributes

text_tokensinteger

audio_tokensinteger

Reserved for multimodal upstreams. Currently always 0; assistant audio bytes are reported in tts.audio_seconds.

reasoning_tokensinteger

Reasoning/thinking tokens emitted by reasoning-capable upstreams.

llmobject

Attribution for the LLM portion of this response.

Show child attributes

modelstring

Effective upstream model ID after router resolution (e.g. openai/gpt-4o-mini).

ttsobject

TTS usage accumulated across all segments of this response.

Show child attributes

modelstring

TTS model used for this response (e.g. inworld-tts-2).

charactersinteger

Total characters synthesized.

audio_secondsnumber

Total assistant audio duration emitted by TTS, in seconds.

sttobject

STT usage for the user audio that fed this response. Drained per response.done from a rolling per-session counter, so each response sees only the audio transcribed since the previous response.done.

Show child attributes

modelstring

STT model used (e.g. inworld/inworld-stt-1).

audio_secondsnumber

User audio duration transcribed for this turn, in seconds.

metadataobject

Metadata echoed from response.create. Omitted when the client did not supply response metadata; never contains LLM Router providerData.metadata.

Example

{
  "event_id": "string",
  "response": {
    "id": "string",
    "object": "string",
    "status": "in_progress",
    "status_details": {
      "type": "completed",
      "reason": "string",
      "error": {
        "type": "string",
        "code": "string"
      }
    },
    "output": [
      {
        "object": "string",
        "id": "string",
        "type": "message",
        "status": "completed",
        "role": "system",
        "content": [
          {
            "type": "input_text",
            "text": "string",
            "audio": "string",
            "transcript": "string",
            "image": "string",
            "image_url": "string",
            "detail": "low"
          }
        ],
        "call_id": "string",
        "name": "string",
        "arguments": "string",
        "output": "string",
        "force_uninterruptible": true
      }
    ],
    "conversation_id": "string",
    "output_modalities": [
      "string"
    ],
    "max_output_tokens": 0,
    "usage": {
      "total_tokens": 0,
      "input_tokens": 0,
      "output_tokens": 0,
      "input_token_details": {
        "cached_tokens": 0,
        "cache_write_tokens": 0,
        "text_tokens": 0,
        "audio_tokens": 0
      },
      "output_token_details": {
        "text_tokens": 0,
        "audio_tokens": 0,
        "reasoning_tokens": 0
      },
      "llm": {
        "model": "string"
      },
      "tts": {
        "model": "string",
        "characters": 0,
        "audio_seconds": 0
      },
      "stt": {
        "model": "string",
        "audio_seconds": 0
      }
    }
  }
}
response.output_item.added

An output item was added to the response.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

response_idstringrequired

The response this item belongs to.

output_indexinteger

Index of the output item in the response.

itemobjectrequired

Server conversation item snapshot. Variant-specific function fields are always serialized, including as empty strings.

Show child attributes

objectstring

Object type identifier (read-only, present in server responses).

idstring

Item ID.

typeenum<string>required

Item variant.

Available options:messagefunction_callfunction_call_output

statusenum<string>

Item status (read-only, present in server responses).

Available options:completedin_progressincomplete

roleenum<string>

Available options:systemuserassistanttool

contentobject[]

Show child attributes

typeenum<string>required

Content type. User input uses inputtext/inputaudio/inputimage. Assistant item snapshots use outputtext/outputaudio. Streaming response.contentpart events use text/audio. Legacy text/audio aliases are accepted when creating assistant items.

Available options:input_textinput_audioinput_imagetextaudiooutput_textoutput_audio

textstring

Text content.

audiostring

Base64-encoded audio.

transcriptstring

Human-readable transcript accompanying audio.

imagestring

Inline image data for input_image content.

image_urlstring

Image URL for input_image content.

detailenum<string>

Image detail level.

Available options:lowhighauto

call_idstring

Tool-call identifier. Required and non-empty for client-created functioncalloutput; optional for functioncall. Always present on server functioncall_output snapshots, including as an empty string.

namestring

Function name. Required and non-empty for client-created functioncall; always present on server functioncall snapshots, including as an empty string.

argumentsstring

Function arguments as a JSON string. May be omitted on input and then normalizes to an empty string; always present on server function_call snapshots.

outputstring

Function result. May be omitted on input and then normalizes to an empty string; always present on server functioncalloutput snapshots.

force_uninterruptibleboolean

Inworld extension (read-only). Present and true on the assistant message produced by a response.speak request that set force_uninterruptible; omitted otherwise. It describes that response only and does not protect later responses.

Example

{
  "event_id": "string",
  "response_id": "string",
  "output_index": 0,
  "item": {
    "object": "string",
    "id": "string",
    "type": "message",
    "status": "completed",
    "role": "system",
    "content": [
      {
        "type": "input_text",
        "text": "string",
        "audio": "string",
        "transcript": "string",
        "image": "string",
        "image_url": "string",
        "detail": "low"
      }
    ],
    "call_id": "string",
    "name": "string",
    "arguments": "string",
    "output": "string",
    "force_uninterruptible": true
  }
}
response.output_item.done

An output item finished.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

response_idstringrequired

The response this item belongs to.

output_indexinteger

Index of the output item in the response.

itemobjectrequired

Server conversation item snapshot. Variant-specific function fields are always serialized, including as empty strings.

Show child attributes

objectstring

Object type identifier (read-only, present in server responses).

idstring

Item ID.

typeenum<string>required

Item variant.

Available options:messagefunction_callfunction_call_output

statusenum<string>

Item status (read-only, present in server responses).

Available options:completedin_progressincomplete

roleenum<string>

Available options:systemuserassistanttool

contentobject[]

Show child attributes

typeenum<string>required

Content type. User input uses inputtext/inputaudio/inputimage. Assistant item snapshots use outputtext/outputaudio. Streaming response.contentpart events use text/audio. Legacy text/audio aliases are accepted when creating assistant items.

Available options:input_textinput_audioinput_imagetextaudiooutput_textoutput_audio

textstring

Text content.

audiostring

Base64-encoded audio.

transcriptstring

Human-readable transcript accompanying audio.

imagestring

Inline image data for input_image content.

image_urlstring

Image URL for input_image content.

detailenum<string>

Image detail level.

Available options:lowhighauto

call_idstring

Tool-call identifier. Required and non-empty for client-created functioncalloutput; optional for functioncall. Always present on server functioncall_output snapshots, including as an empty string.

namestring

Function name. Required and non-empty for client-created functioncall; always present on server functioncall snapshots, including as an empty string.

argumentsstring

Function arguments as a JSON string. May be omitted on input and then normalizes to an empty string; always present on server function_call snapshots.

outputstring

Function result. May be omitted on input and then normalizes to an empty string; always present on server functioncalloutput snapshots.

force_uninterruptibleboolean

Inworld extension (read-only). Present and true on the assistant message produced by a response.speak request that set force_uninterruptible; omitted otherwise. It describes that response only and does not protect later responses.

Example

{
  "event_id": "string",
  "response_id": "string",
  "output_index": 0,
  "item": {
    "object": "string",
    "id": "string",
    "type": "message",
    "status": "completed",
    "role": "system",
    "content": [
      {
        "type": "input_text",
        "text": "string",
        "audio": "string",
        "transcript": "string",
        "image": "string",
        "image_url": "string",
        "detail": "low"
      }
    ],
    "call_id": "string",
    "name": "string",
    "arguments": "string",
    "output": "string",
    "force_uninterruptible": true
  }
}
response.content_part.added

A content part was added to an output item.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

response_idstringrequired

Response identifier.

item_idstring

Item identifier.

output_indexinteger

Index of the output item.

content_indexinteger

Index of the content part.

partobjectrequired

The content part. type: "audio" or "text". transcript: initially empty string.

Example

{
  "event_id": "string",
  "response_id": "string",
  "item_id": "string",
  "output_index": 0,
  "content_index": 0
}
response.content_part.done

A content part finished.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

response_idstringrequired

Response identifier.

item_idstring

Item identifier.

output_indexinteger

Index of the output item.

content_indexinteger

Index of the content part.

partobjectrequired

The completed content part with final transcript.

Example

{
  "event_id": "string",
  "response_id": "string",
  "item_id": "string",
  "output_index": 0,
  "content_index": 0
}
response.output_text.delta

Streaming text chunk from the model.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

response_idstringrequired

Response identifier.

item_idstring

Item identifier.

output_indexinteger

Index of the output item.

content_indexinteger

Index of the content part.

deltastringrequired

Text chunk.

Example

{
  "event_id": "string",
  "response_id": "string",
  "item_id": "string",
  "output_index": 0,
  "content_index": 0,
  "delta": "string"
}
response.output_text.done

Text output finished.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

response_idstringrequired

Response identifier.

item_idstring

Item identifier.

output_indexinteger

Index of the output item.

content_indexinteger

Index of the content part.

textstringrequired

Complete text output.

Example

{
  "event_id": "string",
  "response_id": "string",
  "item_id": "string",
  "output_index": 0,
  "content_index": 0,
  "text": "string"
}
response.output_audio_transcript.delta

Streaming transcript for generated audio.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

response_idstringrequired

Response identifier.

item_idstring

Item identifier.

output_indexinteger

Index of the output item.

content_indexinteger

Index of the content part.

deltastringrequired

Transcript chunk.

Example

{
  "event_id": "string",
  "response_id": "string",
  "item_id": "string",
  "output_index": 0,
  "content_index": 0,
  "delta": "string"
}
response.output_audio_transcript.done

Final transcript for generated audio.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

response_idstringrequired

Response identifier.

item_idstring

Item identifier.

output_indexinteger

Index of the output item.

content_indexinteger

Index of the content part.

transcriptstringrequired

Complete transcript.

Example

{
  "event_id": "string",
  "response_id": "string",
  "item_id": "string",
  "output_index": 0,
  "content_index": 0,
  "transcript": "string"
}
response.output_audio.done

Audio output for a content part finished.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

response_idstringrequired

Response identifier.

item_idstring

Item identifier.

output_indexinteger

Index of the output item.

content_indexinteger

Index of the content part.

Example

{
  "event_id": "string",
  "response_id": "string",
  "item_id": "string",
  "output_index": 0,
  "content_index": 0
}
response.function_call_arguments.delta

Streaming tool call arguments.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

response_idstringrequired

Response identifier.

item_idstringrequired

Item identifier.

output_indexintegerrequired

Index of the output item.

call_idstringrequired

Tool-call identifier.

deltastringrequired

Arguments chunk (JSON string fragment).

Example

{
  "event_id": "string",
  "response_id": "string",
  "item_id": "string",
  "output_index": 0,
  "call_id": "string",
  "delta": "string"
}
response.function_call_arguments.done

Tool call arguments finished.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

response_idstringrequired

Response identifier.

item_idstringrequired

Item identifier.

output_indexintegerrequired

Index of the output item.

call_idstringrequired

Tool-call identifier.

namestringrequired

Function name.

argumentsstringrequired

Complete tool call arguments (JSON string).

Example

{
  "event_id": "string",
  "response_id": "string",
  "item_id": "string",
  "output_index": 0,
  "call_id": "string",
  "name": "string",
  "arguments": "string"
}
input_audio_buffer.speech_started

Voice activity detected — user started speaking. Always emitted before transcripts, including on STT providers without native VAD (the server synthesizes the event at first-audio so client code can rely on the same ordering across providers).

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

audio_start_msinteger

Millisecond offset in the audio stream where speech was detected.

item_idstringrequired

The conversation item ID associated with this speech segment.

Example

{
  "event_id": "string",
  "audio_start_ms": 0,
  "item_id": "string"
}
input_audio_buffer.speech_stopped

Voice activity ended — user stopped speaking.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

audio_end_msinteger

Millisecond offset in the audio stream where speech ended.

item_idstringrequired

The conversation item ID associated with this speech segment.

Example

{
  "event_id": "string",
  "audio_end_ms": 0,
  "item_id": "string"
}
input_audio_buffer.committed

Buffered audio was committed as a conversation item.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

previous_item_idstringnull

The ID of the preceding conversation item, or null.

item_idstringrequired

The new conversation item ID for the committed audio.

Example

{
  "event_id": "string",
  "item_id": "string"
}
input_audio_buffer.cleared

Input audio buffer was cleared.

Payload

typeobjectrequired

event_idstringrequired

Example

{
  "event_id": "string"
}
input_audio_buffer.timeout_triggered

An idle timeout was triggered on the input buffer. Emitted under servervad when no speech has been detected within `turndetection.idletimeoutms`.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

audio_start_msintegerrequired

Audio buffer start offset (ms) at the time the idle timeout fired.

audio_end_msintegerrequired

Audio buffer end offset (ms) at the time the idle timeout fired.

item_idstringrequired

Conversation item ID associated with the idle audio buffer.

Example

{
  "event_id": "string",
  "audio_start_ms": 0,
  "audio_end_ms": 0,
  "item_id": "string"
}
input_audio_buffer.turn_suggestion

Emitted by the server VAD smart-turn detector when it predicts the user has reached an end-of-turn boundary. Clients can use this signal to drive low-latency UI cues or to pre-warm a response without waiting for the final speechstopped commit. May be followed by `inputaudiobuffer.turnsuggestion_revoked` if the user resumes speaking.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

item_idstringrequired

The conversation item ID associated with this utterance.

utterance_indexintegerrequired

Monotonic index of the utterance within the session. Pairs with the matching turn_suggestion_revoked.

probabilitynumberrequired

Smart-turn model end-of-turn probability (0.0–1.0).

trailing_silence_msnumber

Trailing silence at the time of inference, in milliseconds.

audio_duration_msnumber

Audio duration of the utterance at the time of inference, in milliseconds.

inference_msnumber

Smart-turn model inference latency, in milliseconds.

Example

{
  "event_id": "string",
  "item_id": "string",
  "utterance_index": 0,
  "probability": 0,
  "trailing_silence_ms": 0,
  "audio_duration_ms": 0,
  "inference_ms": 0
}
input_audio_buffer.turn_suggestion_revoked

Emitted when the user resumes speaking after a previous turnsuggestion. Pairs with the most recent turnsuggestion sharing the same utterance_index.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

item_idstringrequired

The conversation item ID associated with this utterance.

utterance_indexintegerrequired

Index of the utterance whose previous turn_suggestion is being revoked.

Example

{
  "event_id": "string",
  "item_id": "string",
  "utterance_index": 0
}
output_audio_buffer.started

Server started sending output audio.

Payload

typeobjectrequired

event_idstringrequired

Example

{
  "event_id": "string"
}
output_audio_buffer.stopped

Server stopped sending output audio.

Payload

typeobjectrequired

event_idstringrequired

Example

{
  "event_id": "string"
}
output_audio_buffer.cleared

Output audio buffer was cleared.

Payload

typeobjectrequired

event_idstringrequired

Example

{
  "event_id": "string"
}
response.backchannel.audio.delta

Streaming PCM audio chunk for a low-latency back-channel interjection (e.g. "uh-huh", "right") emitted while the user is mid-utterance. Out-of-band from the main response stream — use backchannel_id to group chunks belonging to the same interjection.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

backchannel_idstringrequired

Synthetic ID grouping deltas + done for a single back-channel interjection. Use as the playback bucket key so chunks of one interjection don't collide with the active response item.

deltastringrequired

Base64-encoded audio chunk in the session's configured audio.output.format (PCM16, audio/pcmu, or audio/pcma).

Example

{
  "event_id": "string",
  "backchannel_id": "string",
  "delta": "string"
}
response.backchannel.audio.done

All audio for a back-channel interjection has been streamed. No teardown required — playback queues until exhausted.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

backchannel_idstringrequired

Identifies which back-channel interjection finished streaming.

phrasestring

The chosen back-channel utterance (e.g. "uh-huh", "right"). Optional — omitted when the decider doesn't surface the phrase to clients.

Example

{
  "event_id": "string",
  "backchannel_id": "string",
  "phrase": "string"
}
response.backchannel.skipped

An evaluation tick chose not to fire a back-channel. Useful for client-side telemetry; clients that don't care can ignore this event.

Payload

typeobjectrequired

event_idstringrequired

Server-generated event ID.

reasonstringrequired

Short machine-readable string describing why no back-channel was emitted on this evaluation tick (e.g. min_gap_not_elapsed, deadline_missed, no_phrase). Stable enough for telemetry; clients that don't care can ignore.

Example

{
  "event_id": "string",
  "reason": "string"
}
rate_limits.updated

Reports current rate limit state.

Payload

typeobjectrequired

event_idstringrequired

Example

{
  "event_id": "string"
}