WebSocket
Realtime API (WebSocket)
wss://api.inworld.ai/api/v1/realtime/sessionReal-time, multimodal AI interactions over WebSocket. Enables low-latency speech-to-speech conversations through a cascaded pipeline (STT → LLM → TTS), supporting both audio and text modalities.
The API maintains a persistent WebSocket connection where clients can:
- Create and configure sessions with custom instructions and voice settings
- Stream audio input in real-time for natural voice conversations
- Send text input as an alternative to audio
- Receive streaming audio and text responses with low latency
- Manage conversation flow with turn detection and response control
Key Features:
- Low Latency: Optimized for real-time interactions
- Multimodal: Supports both audio and text input/output
- Voice Activity Detection: Automatic speech detection with configurable thresholds
- Streaming Responses: Receive response events as they're generated
- Session Management: Maintain conversation context across multiple interactions
Rate Limits: Concurrent session limits vary by subscription plan. See features and limits by plan for the per-tier table.
Inworld extensions: The session object accepts a providerData field carrying Inworld-specific extensions to the OpenAI-compatible shape — STT tuning, TTS segmentation/steering, automatic memory, back-channel, and responsiveness fillers. See API Extensions for the field-by-field reference.
This API implements the Realtime interface. Refer to the Realtime overview for hands-on guides.
Client messages
session.update
Update the session configuration. The server responds with a session.updated event.
Payload
typeobjectrequired
event_idstring
Optional client-generated event ID.
sessionobjectrequired
The session object configures model behavior, audio settings, tools, and more. It appears in session.update, session.created, and session.updated events.
Show child attributes
objectstring
Object type identifier (read-only).
typestring
Fixed value.
idstring
Server-assigned session ID (read-only).
modelstring
Model identifier.
instructionsstring
System instructions for the model.
output_modalitiesenum<string>[]
Output types: "text", "audio", or both.
temperaturenumber
The sampling temperature used for response generation.
max_output_tokensoneOf
Maximum tokens for a response. Either an integer (1–4096) or the literal string "inf".
audioobject
Show child attributes
inputobject
Show child attributes
formatoneOf
Audio wire format for client input or server output. Used on both audio.input.format and audio.output.format. Accepts either an object form (with type + optional rate) or a legacy string alias.
noise_reductionobject
Show child attributes
typeenum<string>required
Noise reduction mode.
Available options:near_fieldfar_field
transcriptionobject
Show child attributes
modelstring
Transcription model identifier (e.g., inworld/inworld-stt-1).
languagestring
Optional language code.
promptstring
Not supported by Inworld STT. Use audio.input.transcription.prompts instead.
promptsstring[]
Expected words or phrases to help Inworld STT recognize names and domain-specific vocabulary.
turn_detectiononeOf
outputobject
Show child attributes
formatoneOf
Audio wire format for client input or server output. Used on both audio.input.format and audio.output.format. Accepts either an object form (with type + optional rate) or a legacy string alias.
voicestring
Voice preset for audio output (e.g., Dennis). See the List Voices API or the Voice library page in the Inworld Portal for the full list of supported voices.
modelstring
The TTS model used for audio output.
speednumber
Playback speed (0.25–1.5).
toolsobject[]
Show child attributes
typestringrequired
Tool type.
namestringrequired
Function name.
descriptionstring
What the function does.
parametersobject
JSON Schema for function parameters.
tool_choiceoneOf
Tool selection. Either a shorthand string (none, auto, required) or a ToolChoiceTarget object that names a specific function or MCP server.
truncationoneOf
Conversation truncation strategy. Either a shorthand string (auto, disabled) or a RetentionRatioTruncation object that pins a fraction of the context window.
tracingoneOf
Tracing configuration. Either the literal "auto" (server defaults) or a TracingConfig object with explicit workflow / group / metadata.
includeenum<string>[]
Optional data to include.
promptstringnull
Prompt template reference, or null.
providerDataobject
Inworld-specific extensions to the OpenAI-compatible session shape. Most fields are hot-swappable via partial session.update. See Inworld Realtime API Extensions for the field-by-field reference.
Show child attributes
sttobject
STT extensions: voice_profile, end_of_turn_confidence_threshold, vad_threshold, min_end_of_turn_silence, max_turn_silence. See STT provider data.
ttsobject
TTS segmentation, language, delivery, and alignment controls: segmenter_strategy, steering_handling, language, delivery_mode, conversational, user_turn_mode, timestamp_type, timestamp_transport_strategy. See TTS provider data.
memoryobject
Automatic conversation memory and summarization: enabled, turn_interval, max_memory_length, max_transcript_items, max_facts, trim_after_summarize. See Memory provider data.
backchannelobject
Opt-in back-channel responses (brief acknowledgements while the user is speaking). See Back-channel.
responsivenessobject
Opt-in responsiveness fillers (low-latency acknowledgements emitted before the main response). See Responsiveness.
auto_tool_responsebooleandefault: true
When true, adding a functioncalloutput automatically starts the follow-up response. Set false for OpenAI-compatible client-controlled continuation, then send response.create explicitly. Omitted partial updates preserve the current value.
text_generation_configobject
Fine-grained LLM generation parameters. Also accepted under providerData.text_generation_config.
Show child attributes
reasoningobject
Chain-of-thought reasoning control. Forwarded to the LLM Router as extra_body.reasoning.
Show child attributes
effortenum<string>
Reasoning depth. Higher values allocate more thinking tokens.
Available options:NONEMINIMALLOWMEDIUMHIGHXHIGH
maxTokensinteger
Cap on reasoning/thinking tokens.
excludeboolean
When true, reasoning tokens are generated but excluded from response text.
temperaturenumber
Sampling temperature override.
topPnumber
Nucleus sampling.
maxNewTokensinteger
Max completion tokens.
frequencyPenaltynumber
Frequency penalty.
presencePenaltynumber
Presence penalty.
expires_atinteger
Unix timestamp for session expiration (read-only).
Example
{
"type": "session.update",
"session": {
"instructions": "You are a friendly voice assistant.",
"audio": {
"input": {
"transcription": {
"model": "inworld/inworld-stt-1"
},
"turn_detection": {
"type": "semantic_vad",
"eagerness": "medium",
"create_response": true,
"interrupt_response": true
}
},
"output": {
"model": "inworld-tts-2",
"voice": "Dennis",
"speed": 1
}
}
}
}conversation.item.create
Add a conversation item (message, function call, or function call output).
Payload
typeobjectrequired
event_idstring
Optional client-generated event ID.
previous_item_idstring
Controls insertion. Omit to append, use "root" to prepend, or provide an existing item ID to insert immediately after it. An unknown ID returns itemnotfound with error.param set to previousitemid and does not mutate history.
itemobjectrequired
Client-created conversation item. functioncall requires a non-empty name; functioncalloutput requires a non-empty callid. arguments and output may be omitted and normalize to empty strings.
Show child attributes
objectstring
Object type identifier (read-only, present in server responses).
idstring
Item ID.
typeenum<string>required
Item variant.
Available options:messagefunction_callfunction_call_output
statusenum<string>
Item status (read-only, present in server responses).
Available options:completedin_progressincomplete
roleenum<string>
Available options:systemuserassistanttool
contentobject[]
Show child attributes
typeenum<string>required
Content type. User input uses inputtext/inputaudio/inputimage. Assistant item snapshots use outputtext/outputaudio. Streaming response.contentpart events use text/audio. Legacy text/audio aliases are accepted when creating assistant items.
Available options:input_textinput_audioinput_imagetextaudiooutput_textoutput_audio
textstring
Text content.
audiostring
Base64-encoded audio.
transcriptstring
Human-readable transcript accompanying audio.
imagestring
Inline image data for input_image content.
image_urlstring
Image URL for input_image content.
detailenum<string>
Image detail level.
Available options:lowhighauto
call_idstring
Tool-call identifier. Required and non-empty for client-created functioncalloutput; optional for functioncall. Always present on server functioncall_output snapshots, including as an empty string.
namestring
Function name. Required and non-empty for client-created functioncall; always present on server functioncall snapshots, including as an empty string.
argumentsstring
Function arguments as a JSON string. May be omitted on input and then normalizes to an empty string; always present on server function_call snapshots.
outputstring
Function result. May be omitted on input and then normalizes to an empty string; always present on server functioncalloutput snapshots.
Example
{
"type": "conversation.item.create",
"item": {
"type": "message",
"role": "user",
"content": [
{
"type": "input_text",
"text": "Hello, how are you?"
}
]
}
}conversation.item.truncate
Truncate an assistant message's audio.
Payload
typeobjectrequired
event_idstring
Optional client-generated event ID.
item_idstringrequired
The ID of the assistant message item to truncate.
content_indexintegerrequired
Index of the content part to truncate.
audio_end_msintegerrequired
Millisecond offset to truncate the audio at.
Example
{
"event_id": "string",
"item_id": "string",
"content_index": 0,
"audio_end_ms": 0
}conversation.item.delete
Delete a conversation item by ID.
Payload
typeobjectrequired
event_idstring
Optional client-generated event ID.
item_idstringrequired
The ID of the conversation item to delete.
Example
{
"event_id": "string",
"item_id": "string"
}conversation.item.retrieve
Retrieve a conversation item by ID.
Payload
typeobjectrequired
event_idstring
Optional client-generated event ID.
item_idstringrequired
The ID of the conversation item to retrieve.
Example
{
"event_id": "string",
"item_id": "string"
}response.create
Trigger a model response. The server streams back response events.
Payload
typeobjectrequired
event_idstring
Optional client-generated event ID.
responseobject
Per-response overrides for session defaults.
Show child attributes
conversationoneOf
Conversation selector. Either the literal "auto" (use the session's implicit conversation) or an explicit conversation ID string.
output_modalitiesenum<string>[]
"text", "audio", or both.
instructionsstring
Override instructions for this response.
voicestring
Override voice for this response.
max_output_tokensoneOf
Maximum tokens for a response. Either an integer (1–4096) or the literal string "inf".
tool_choiceoneOf
Tool selection. Either a shorthand string (none, auto, required) or a ToolChoiceTarget object that names a specific function or MCP server.
toolsobject[]
Override available tools.
Show child attributes
typestringrequired
Tool type.
namestringrequired
Function name.
descriptionstring
What the function does.
parametersobject
JSON Schema for function parameters.
metadataobject
Client correlation metadata echoed on response.created and response.done. Keys and values are strings; up to 16 entries, 64 Unicode code points per key, and 512 per value.
Example
{
"type": "response.create",
"response": {
"output_modalities": [
"audio",
"text"
],
"instructions": "Respond in a cheerful tone."
}
}response.cancel
Cancel an in-progress response.
Payload
typeobjectrequired
event_idstring
Optional client-generated event ID.
response_idstring
Cancel a specific response by ID. If omitted, cancels the active response.
Example
{
"event_id": "string",
"response_id": "string"
}input_audio_buffer.append
Append audio bytes to the input buffer.
Payload
typeobjectrequired
event_idstring
Optional client-generated event ID.
audiostringrequired
Base64-encoded audio chunk (~100–200ms) matching the configured input format.
Example
{
"event_id": "string",
"audio": "string"
}input_audio_buffer.commit
Commit the buffered audio as a user message.
Payload
typeobjectrequired
event_idstring
Example
{
"event_id": "string"
}input_audio_buffer.clear
Discard all audio in the input buffer.
Payload
typeobjectrequired
event_idstring
Example
{
"event_id": "string"
}output_audio_buffer.clear
Clear the server's output audio buffer, stopping playback.
Payload
typeobjectrequired
event_idstring
Example
{
"event_id": "string"
}Server messages
session.created
Sent by the server immediately when the WebSocket connection is established, carrying the session's default configuration. Send a session.update to configure the session.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
sessionobjectrequired
The session object configures model behavior, audio settings, tools, and more. It appears in session.update, session.created, and session.updated events.
Show child attributes
objectstring
Object type identifier (read-only).
typestring
Fixed value.
idstring
Server-assigned session ID (read-only).
modelstring
Model identifier.
instructionsstring
System instructions for the model.
output_modalitiesenum<string>[]
Output types: "text", "audio", or both.
temperaturenumber
The sampling temperature used for response generation.
max_output_tokensoneOf
Maximum tokens for a response. Either an integer (1–4096) or the literal string "inf".
audioobject
Show child attributes
inputobject
Show child attributes
formatoneOf
Audio wire format for client input or server output. Used on both audio.input.format and audio.output.format. Accepts either an object form (with type + optional rate) or a legacy string alias.
noise_reductionobject
Show child attributes
typeenum<string>required
Noise reduction mode.
Available options:near_fieldfar_field
transcriptionobject
Show child attributes
modelstring
Transcription model identifier (e.g., inworld/inworld-stt-1).
languagestring
Optional language code.
promptstring
Not supported by Inworld STT. Use audio.input.transcription.prompts instead.
promptsstring[]
Expected words or phrases to help Inworld STT recognize names and domain-specific vocabulary.
turn_detectiononeOf
outputobject
Show child attributes
formatoneOf
Audio wire format for client input or server output. Used on both audio.input.format and audio.output.format. Accepts either an object form (with type + optional rate) or a legacy string alias.
voicestring
Voice preset for audio output (e.g., Dennis). See the List Voices API or the Voice library page in the Inworld Portal for the full list of supported voices.
modelstring
The TTS model used for audio output.
speednumber
Playback speed (0.25–1.5).
toolsobject[]
Show child attributes
typestringrequired
Tool type.
namestringrequired
Function name.
descriptionstring
What the function does.
parametersobject
JSON Schema for function parameters.
tool_choiceoneOf
Tool selection. Either a shorthand string (none, auto, required) or a ToolChoiceTarget object that names a specific function or MCP server.
truncationoneOf
Conversation truncation strategy. Either a shorthand string (auto, disabled) or a RetentionRatioTruncation object that pins a fraction of the context window.
tracingoneOf
Tracing configuration. Either the literal "auto" (server defaults) or a TracingConfig object with explicit workflow / group / metadata.
includeenum<string>[]
Optional data to include.
promptstringnull
Prompt template reference, or null.
providerDataobject
Inworld-specific extensions to the OpenAI-compatible session shape. Most fields are hot-swappable via partial session.update. See Inworld Realtime API Extensions for the field-by-field reference.
Show child attributes
sttobject
STT extensions: voice_profile, end_of_turn_confidence_threshold, vad_threshold, min_end_of_turn_silence, max_turn_silence. See STT provider data.
ttsobject
TTS segmentation, language, delivery, and alignment controls: segmenter_strategy, steering_handling, language, delivery_mode, conversational, user_turn_mode, timestamp_type, timestamp_transport_strategy. See TTS provider data.
memoryobject
Automatic conversation memory and summarization: enabled, turn_interval, max_memory_length, max_transcript_items, max_facts, trim_after_summarize. See Memory provider data.
backchannelobject
Opt-in back-channel responses (brief acknowledgements while the user is speaking). See Back-channel.
responsivenessobject
Opt-in responsiveness fillers (low-latency acknowledgements emitted before the main response). See Responsiveness.
auto_tool_responsebooleandefault: true
When true, adding a functioncalloutput automatically starts the follow-up response. Set false for OpenAI-compatible client-controlled continuation, then send response.create explicitly. Omitted partial updates preserve the current value.
text_generation_configobject
Fine-grained LLM generation parameters. Also accepted under providerData.text_generation_config.
Show child attributes
reasoningobject
Chain-of-thought reasoning control. Forwarded to the LLM Router as extra_body.reasoning.
Show child attributes
effortenum<string>
Reasoning depth. Higher values allocate more thinking tokens.
Available options:NONEMINIMALLOWMEDIUMHIGHXHIGH
maxTokensinteger
Cap on reasoning/thinking tokens.
excludeboolean
When true, reasoning tokens are generated but excluded from response text.
temperaturenumber
Sampling temperature override.
topPnumber
Nucleus sampling.
maxNewTokensinteger
Max completion tokens.
frequencyPenaltynumber
Frequency penalty.
presencePenaltynumber
Presence penalty.
expires_atinteger
Unix timestamp for session expiration (read-only).
Example
{
"event_id": "string",
"session": {
"object": "string",
"type": "string",
"id": "string",
"model": "string",
"instructions": "string",
"output_modalities": [
"text"
],
"temperature": 0,
"max_output_tokens": 0,
"audio": {
"input": {
"format": {
"type": "audio/pcm",
"rate": 0
},
"noise_reduction": {
"type": "near_field"
},
"transcription": {
"model": "string",
"language": "string",
"prompt": "string",
"prompts": [
"string"
]
},
"turn_detection": {
"threshold": 0,
"prefix_padding_ms": 0,
"silence_duration_ms": 0,
"create_response": true,
"interrupt_response": true,
"idle_timeout_ms": 0
}
},
"output": {
"format": {
"type": "audio/pcm",
"rate": 0
},
"voice": "string",
"model": "string",
"speed": 0
}
},
"tools": [
{
"type": "string",
"name": "string",
"description": "string"
}
],
"tool_choice": "none",
"truncation": "auto",
"tracing": "string",
"include": [
"item.input_audio_transcription.logprobs"
],
"providerData": {
"auto_tool_response": true
},
"text_generation_config": {
"reasoning": {
"effort": "NONE",
"maxTokens": 0,
"exclude": true
},
"temperature": 0,
"topP": 0,
"maxNewTokens": 0,
"frequencyPenalty": 0,
"presencePenalty": 0
},
"expires_at": 0
}
}session.updated
Confirms a session.update was applied.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
sessionobjectrequired
The session object configures model behavior, audio settings, tools, and more. It appears in session.update, session.created, and session.updated events.
Show child attributes
objectstring
Object type identifier (read-only).
typestring
Fixed value.
idstring
Server-assigned session ID (read-only).
modelstring
Model identifier.
instructionsstring
System instructions for the model.
output_modalitiesenum<string>[]
Output types: "text", "audio", or both.
temperaturenumber
The sampling temperature used for response generation.
max_output_tokensoneOf
Maximum tokens for a response. Either an integer (1–4096) or the literal string "inf".
audioobject
Show child attributes
inputobject
Show child attributes
formatoneOf
Audio wire format for client input or server output. Used on both audio.input.format and audio.output.format. Accepts either an object form (with type + optional rate) or a legacy string alias.
noise_reductionobject
Show child attributes
typeenum<string>required
Noise reduction mode.
Available options:near_fieldfar_field
transcriptionobject
Show child attributes
modelstring
Transcription model identifier (e.g., inworld/inworld-stt-1).
languagestring
Optional language code.
promptstring
Not supported by Inworld STT. Use audio.input.transcription.prompts instead.
promptsstring[]
Expected words or phrases to help Inworld STT recognize names and domain-specific vocabulary.
turn_detectiononeOf
outputobject
Show child attributes
formatoneOf
Audio wire format for client input or server output. Used on both audio.input.format and audio.output.format. Accepts either an object form (with type + optional rate) or a legacy string alias.
voicestring
Voice preset for audio output (e.g., Dennis). See the List Voices API or the Voice library page in the Inworld Portal for the full list of supported voices.
modelstring
The TTS model used for audio output.
speednumber
Playback speed (0.25–1.5).
toolsobject[]
Show child attributes
typestringrequired
Tool type.
namestringrequired
Function name.
descriptionstring
What the function does.
parametersobject
JSON Schema for function parameters.
tool_choiceoneOf
Tool selection. Either a shorthand string (none, auto, required) or a ToolChoiceTarget object that names a specific function or MCP server.
truncationoneOf
Conversation truncation strategy. Either a shorthand string (auto, disabled) or a RetentionRatioTruncation object that pins a fraction of the context window.
tracingoneOf
Tracing configuration. Either the literal "auto" (server defaults) or a TracingConfig object with explicit workflow / group / metadata.
includeenum<string>[]
Optional data to include.
promptstringnull
Prompt template reference, or null.
providerDataobject
Inworld-specific extensions to the OpenAI-compatible session shape. Most fields are hot-swappable via partial session.update. See Inworld Realtime API Extensions for the field-by-field reference.
Show child attributes
sttobject
STT extensions: voice_profile, end_of_turn_confidence_threshold, vad_threshold, min_end_of_turn_silence, max_turn_silence. See STT provider data.
ttsobject
TTS segmentation, language, delivery, and alignment controls: segmenter_strategy, steering_handling, language, delivery_mode, conversational, user_turn_mode, timestamp_type, timestamp_transport_strategy. See TTS provider data.
memoryobject
Automatic conversation memory and summarization: enabled, turn_interval, max_memory_length, max_transcript_items, max_facts, trim_after_summarize. See Memory provider data.
backchannelobject
Opt-in back-channel responses (brief acknowledgements while the user is speaking). See Back-channel.
responsivenessobject
Opt-in responsiveness fillers (low-latency acknowledgements emitted before the main response). See Responsiveness.
auto_tool_responsebooleandefault: true
When true, adding a functioncalloutput automatically starts the follow-up response. Set false for OpenAI-compatible client-controlled continuation, then send response.create explicitly. Omitted partial updates preserve the current value.
text_generation_configobject
Fine-grained LLM generation parameters. Also accepted under providerData.text_generation_config.
Show child attributes
reasoningobject
Chain-of-thought reasoning control. Forwarded to the LLM Router as extra_body.reasoning.
Show child attributes
effortenum<string>
Reasoning depth. Higher values allocate more thinking tokens.
Available options:NONEMINIMALLOWMEDIUMHIGHXHIGH
maxTokensinteger
Cap on reasoning/thinking tokens.
excludeboolean
When true, reasoning tokens are generated but excluded from response text.
temperaturenumber
Sampling temperature override.
topPnumber
Nucleus sampling.
maxNewTokensinteger
Max completion tokens.
frequencyPenaltynumber
Frequency penalty.
presencePenaltynumber
Presence penalty.
expires_atinteger
Unix timestamp for session expiration (read-only).
Example
{
"event_id": "string",
"session": {
"object": "string",
"type": "string",
"id": "string",
"model": "string",
"instructions": "string",
"output_modalities": [
"text"
],
"temperature": 0,
"max_output_tokens": 0,
"audio": {
"input": {
"format": {
"type": "audio/pcm",
"rate": 0
},
"noise_reduction": {
"type": "near_field"
},
"transcription": {
"model": "string",
"language": "string",
"prompt": "string",
"prompts": [
"string"
]
},
"turn_detection": {
"threshold": 0,
"prefix_padding_ms": 0,
"silence_duration_ms": 0,
"create_response": true,
"interrupt_response": true,
"idle_timeout_ms": 0
}
},
"output": {
"format": {
"type": "audio/pcm",
"rate": 0
},
"voice": "string",
"model": "string",
"speed": 0
}
},
"tools": [
{
"type": "string",
"name": "string",
"description": "string"
}
],
"tool_choice": "none",
"truncation": "auto",
"tracing": "string",
"include": [
"item.input_audio_transcription.logprobs"
],
"providerData": {
"auto_tool_response": true
},
"text_generation_config": {
"reasoning": {
"effort": "NONE",
"maxTokens": 0,
"exclude": true
},
"temperature": 0,
"topP": 0,
"maxNewTokens": 0,
"frequencyPenalty": 0,
"presencePenalty": 0
},
"expires_at": 0
}
}error
Indicates an error occurred.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
errorobjectrequired
Show child attributes
typestringrequired
Error category.
codestring
Error code.
messagestringrequired
Human-readable error description.
paramstring
Related parameter, if applicable.
event_idstring
The client event ID that caused the error, if applicable.
Example
{
"event_id": "2c23cfd4-a4b5-4a96-83b8-a6a151f3989e",
"type": "error",
"error": {
"type": "server_error",
"code": null,
"message": "Failed to read content stream.",
"param": null,
"event_id": null
}
}conversation.item.added
A new item was added to the conversation.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
previous_item_idstringnull
The ID of the preceding conversation item, or null.
itemobjectrequired
Server conversation item snapshot. Variant-specific function fields are always serialized, including as empty strings.
Show child attributes
objectstring
Object type identifier (read-only, present in server responses).
idstring
Item ID.
typeenum<string>required
Item variant.
Available options:messagefunction_callfunction_call_output
statusenum<string>
Item status (read-only, present in server responses).
Available options:completedin_progressincomplete
roleenum<string>
Available options:systemuserassistanttool
contentobject[]
Show child attributes
typeenum<string>required
Content type. User input uses inputtext/inputaudio/inputimage. Assistant item snapshots use outputtext/outputaudio. Streaming response.contentpart events use text/audio. Legacy text/audio aliases are accepted when creating assistant items.
Available options:input_textinput_audioinput_imagetextaudiooutput_textoutput_audio
textstring
Text content.
audiostring
Base64-encoded audio.
transcriptstring
Human-readable transcript accompanying audio.
imagestring
Inline image data for input_image content.
image_urlstring
Image URL for input_image content.
detailenum<string>
Image detail level.
Available options:lowhighauto
call_idstring
Tool-call identifier. Required and non-empty for client-created functioncalloutput; optional for functioncall. Always present on server functioncall_output snapshots, including as an empty string.
namestring
Function name. Required and non-empty for client-created functioncall; always present on server functioncall snapshots, including as an empty string.
argumentsstring
Function arguments as a JSON string. May be omitted on input and then normalizes to an empty string; always present on server function_call snapshots.
outputstring
Function result. May be omitted on input and then normalizes to an empty string; always present on server functioncalloutput snapshots.
Example
{
"event_id": "string",
"item": {
"object": "string",
"id": "string",
"type": "message",
"status": "completed",
"role": "system",
"content": [
{
"type": "input_text",
"text": "string",
"audio": "string",
"transcript": "string",
"image": "string",
"image_url": "string",
"detail": "low"
}
],
"call_id": "string",
"name": "string",
"arguments": "string",
"output": "string"
}
}conversation.item.done
An item finished being populated.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
previous_item_idstringnull
The ID of the preceding conversation item, or null.
itemobjectrequired
Server conversation item snapshot. Variant-specific function fields are always serialized, including as empty strings.
Show child attributes
objectstring
Object type identifier (read-only, present in server responses).
idstring
Item ID.
typeenum<string>required
Item variant.
Available options:messagefunction_callfunction_call_output
statusenum<string>
Item status (read-only, present in server responses).
Available options:completedin_progressincomplete
roleenum<string>
Available options:systemuserassistanttool
contentobject[]
Show child attributes
typeenum<string>required
Content type. User input uses inputtext/inputaudio/inputimage. Assistant item snapshots use outputtext/outputaudio. Streaming response.contentpart events use text/audio. Legacy text/audio aliases are accepted when creating assistant items.
Available options:input_textinput_audioinput_imagetextaudiooutput_textoutput_audio
textstring
Text content.
audiostring
Base64-encoded audio.
transcriptstring
Human-readable transcript accompanying audio.
imagestring
Inline image data for input_image content.
image_urlstring
Image URL for input_image content.
detailenum<string>
Image detail level.
Available options:lowhighauto
call_idstring
Tool-call identifier. Required and non-empty for client-created functioncalloutput; optional for functioncall. Always present on server functioncall_output snapshots, including as an empty string.
namestring
Function name. Required and non-empty for client-created functioncall; always present on server functioncall snapshots, including as an empty string.
argumentsstring
Function arguments as a JSON string. May be omitted on input and then normalizes to an empty string; always present on server function_call snapshots.
outputstring
Function result. May be omitted on input and then normalizes to an empty string; always present on server functioncalloutput snapshots.
Example
{
"event_id": "string",
"item": {
"object": "string",
"id": "string",
"type": "message",
"status": "completed",
"role": "system",
"content": [
{
"type": "input_text",
"text": "string",
"audio": "string",
"transcript": "string",
"image": "string",
"image_url": "string",
"detail": "low"
}
],
"call_id": "string",
"name": "string",
"arguments": "string",
"output": "string"
}
}conversation.item.deleted
An item was deleted from the conversation.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
item_idstringrequired
The ID of the deleted item.
Example
{
"event_id": "string",
"item_id": "string"
}conversation.item.retrieved
Response to conversation.item.retrieve.
Payload
typeobjectrequired
event_idstringrequired
Example
{
"event_id": "string"
}conversation.item.truncated
An assistant audio item was truncated.
Payload
typeobjectrequired
event_idstringrequired
Example
{
"event_id": "string"
}conversation.item.input_audio_transcription.delta
Streaming partial transcription for user audio.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
item_idstringrequired
The conversation item being transcribed.
content_indexinteger
Index of the content part being transcribed.
deltastringrequired
Partial transcription text.
Example
{
"event_id": "string",
"item_id": "string",
"content_index": 0,
"delta": "string"
}conversation.item.input_audio_transcription.completed
Final transcription for a user audio item.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
item_idstringrequired
The conversation item that was transcribed.
content_indexinteger
Index of the content part that was transcribed.
transcriptstringrequired
Complete transcription text.
Example
{
"event_id": "string",
"item_id": "string",
"content_index": 0,
"transcript": "string"
}response.created
A new response was created. Contains the full response object in its initial state.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
responseobjectrequired
The response object returned in response.created and response.done events.
Show child attributes
idstring
Response identifier.
objectstring
Object type.
statusenum<string>
Response status.
Available options:in_progresscompletedcancelledfailed
status_detailsoneOf
Status detail object for the response, or null if no details are available.
outputobject[]
Output items.
Show child attributes
objectstring
Object type identifier (read-only, present in server responses).
idstring
Item ID.
typeenum<string>required
Item variant.
Available options:messagefunction_callfunction_call_output
statusenum<string>
Item status (read-only, present in server responses).
Available options:completedin_progressincomplete
roleenum<string>
Available options:systemuserassistanttool
contentobject[]
Show child attributes
typeenum<string>required
Content type. User input uses inputtext/inputaudio/inputimage. Assistant item snapshots use outputtext/outputaudio. Streaming response.contentpart events use text/audio. Legacy text/audio aliases are accepted when creating assistant items.
Available options:input_textinput_audioinput_imagetextaudiooutput_textoutput_audio
textstring
Text content.
audiostring
Base64-encoded audio.
transcriptstring
Human-readable transcript accompanying audio.
imagestring
Inline image data for input_image content.
image_urlstring
Image URL for input_image content.
detailenum<string>
Image detail level.
Available options:lowhighauto
call_idstring
Tool-call identifier. Required and non-empty for client-created functioncalloutput; optional for functioncall. Always present on server functioncall_output snapshots, including as an empty string.
namestring
Function name. Required and non-empty for client-created functioncall; always present on server functioncall snapshots, including as an empty string.
argumentsstring
Function arguments as a JSON string. May be omitted on input and then normalizes to an empty string; always present on server function_call snapshots.
outputstring
Function result. May be omitted on input and then normalizes to an empty string; always present on server functioncalloutput snapshots.
conversation_idstring
Conversation this response belongs to.
output_modalitiesstring[]
"text", "audio", or both.
max_output_tokensoneOf
Maximum tokens for a response. Either an integer (1–4096) or the literal string "inf".
audioobject
Audio output config echoed from session.
instructionsstring
Instructions applied to this response (echoed from session).
usageobject
Token + audio usage for the response. The base block (total_tokens, input_tokens, output_tokens, input_token_details, output_token_details) covers LLM token accounting. The optional llm, tts, and stt sub-objects attribute usage per modality to the upstream model/provider and add tts.audio_seconds/stt.audio_seconds for audio billing. Each modality sub-object is omitted when its data is unavailable (e.g. a response with no TTS output, or a TTS-only response with no STT turn).
Show child attributes
total_tokensinteger
input_tokensinteger
output_tokensinteger
input_token_detailsobject
Show child attributes
cached_tokensinteger
Input tokens served from a prompt cache hit. Populated by both implicit provider caching and explicit caching via providerData.caching.
cache_write_tokensinteger
Input tokens written to the cache when establishing a new entry (explicit caching on supporting providers, e.g. Anthropic). Omitted on cache hits.
text_tokensinteger
audio_tokensinteger
Reserved for multimodal upstreams. Currently always 0 on Inworld's router-fronted models.
output_token_detailsobject
Show child attributes
text_tokensinteger
audio_tokensinteger
Reserved for multimodal upstreams. Currently always 0; assistant audio bytes are reported in tts.audio_seconds.
reasoning_tokensinteger
Reasoning/thinking tokens emitted by reasoning-capable upstreams.
llmobject
Attribution for the LLM portion of this response.
Show child attributes
modelstring
Effective upstream model ID after router resolution (e.g. openai/gpt-4o-mini).
ttsobject
TTS usage accumulated across all segments of this response.
Show child attributes
modelstring
TTS model used for this response (e.g. inworld-tts-2).
charactersinteger
Total characters synthesized.
audio_secondsnumber
Total assistant audio duration emitted by TTS, in seconds.
sttobject
STT usage for the user audio that fed this response. Drained per response.done from a rolling per-session counter, so each response sees only the audio transcribed since the previous response.done.
Show child attributes
modelstring
STT model used (e.g. inworld/inworld-stt-1).
audio_secondsnumber
User audio duration transcribed for this turn, in seconds.
metadataobject
Metadata echoed from response.create. Omitted when the client did not supply response metadata; never contains LLM Router providerData.metadata.
Example
{
"event_id": "string",
"response": {
"id": "string",
"object": "string",
"status": "in_progress",
"status_details": {
"type": "completed",
"reason": "string",
"error": {
"type": "string",
"code": "string"
}
},
"output": [
{
"object": "string",
"id": "string",
"type": "message",
"status": "completed",
"role": "system",
"content": [
{
"type": "input_text",
"text": "string",
"audio": "string",
"transcript": "string",
"image": "string",
"image_url": "string",
"detail": "low"
}
],
"call_id": "string",
"name": "string",
"arguments": "string",
"output": "string"
}
],
"conversation_id": "string",
"output_modalities": [
"string"
],
"max_output_tokens": 0,
"instructions": "string",
"usage": {
"total_tokens": 0,
"input_tokens": 0,
"output_tokens": 0,
"input_token_details": {
"cached_tokens": 0,
"cache_write_tokens": 0,
"text_tokens": 0,
"audio_tokens": 0
},
"output_token_details": {
"text_tokens": 0,
"audio_tokens": 0,
"reasoning_tokens": 0
},
"llm": {
"model": "string"
},
"tts": {
"model": "string",
"characters": 0,
"audio_seconds": 0
},
"stt": {
"model": "string",
"audio_seconds": 0
}
}
}
}response.done
The response finished. Contains the completed response object with final status and output.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
responseobjectrequired
The response object returned in response.created and response.done events.
Show child attributes
idstring
Response identifier.
objectstring
Object type.
statusenum<string>
Response status.
Available options:in_progresscompletedcancelledfailed
status_detailsoneOf
Status detail object for the response, or null if no details are available.
outputobject[]
Output items.
Show child attributes
objectstring
Object type identifier (read-only, present in server responses).
idstring
Item ID.
typeenum<string>required
Item variant.
Available options:messagefunction_callfunction_call_output
statusenum<string>
Item status (read-only, present in server responses).
Available options:completedin_progressincomplete
roleenum<string>
Available options:systemuserassistanttool
contentobject[]
Show child attributes
typeenum<string>required
Content type. User input uses inputtext/inputaudio/inputimage. Assistant item snapshots use outputtext/outputaudio. Streaming response.contentpart events use text/audio. Legacy text/audio aliases are accepted when creating assistant items.
Available options:input_textinput_audioinput_imagetextaudiooutput_textoutput_audio
textstring
Text content.
audiostring
Base64-encoded audio.
transcriptstring
Human-readable transcript accompanying audio.
imagestring
Inline image data for input_image content.
image_urlstring
Image URL for input_image content.
detailenum<string>
Image detail level.
Available options:lowhighauto
call_idstring
Tool-call identifier. Required and non-empty for client-created functioncalloutput; optional for functioncall. Always present on server functioncall_output snapshots, including as an empty string.
namestring
Function name. Required and non-empty for client-created functioncall; always present on server functioncall snapshots, including as an empty string.
argumentsstring
Function arguments as a JSON string. May be omitted on input and then normalizes to an empty string; always present on server function_call snapshots.
outputstring
Function result. May be omitted on input and then normalizes to an empty string; always present on server functioncalloutput snapshots.
conversation_idstring
Conversation this response belongs to.
output_modalitiesstring[]
"text", "audio", or both.
max_output_tokensoneOf
Maximum tokens for a response. Either an integer (1–4096) or the literal string "inf".
audioobject
Audio output config echoed from session.
instructionsstring
Instructions applied to this response (echoed from session).
usageobject
Token + audio usage for the response. The base block (total_tokens, input_tokens, output_tokens, input_token_details, output_token_details) covers LLM token accounting. The optional llm, tts, and stt sub-objects attribute usage per modality to the upstream model/provider and add tts.audio_seconds/stt.audio_seconds for audio billing. Each modality sub-object is omitted when its data is unavailable (e.g. a response with no TTS output, or a TTS-only response with no STT turn).
Show child attributes
total_tokensinteger
input_tokensinteger
output_tokensinteger
input_token_detailsobject
Show child attributes
cached_tokensinteger
Input tokens served from a prompt cache hit. Populated by both implicit provider caching and explicit caching via providerData.caching.
cache_write_tokensinteger
Input tokens written to the cache when establishing a new entry (explicit caching on supporting providers, e.g. Anthropic). Omitted on cache hits.
text_tokensinteger
audio_tokensinteger
Reserved for multimodal upstreams. Currently always 0 on Inworld's router-fronted models.
output_token_detailsobject
Show child attributes
text_tokensinteger
audio_tokensinteger
Reserved for multimodal upstreams. Currently always 0; assistant audio bytes are reported in tts.audio_seconds.
reasoning_tokensinteger
Reasoning/thinking tokens emitted by reasoning-capable upstreams.
llmobject
Attribution for the LLM portion of this response.
Show child attributes
modelstring
Effective upstream model ID after router resolution (e.g. openai/gpt-4o-mini).
ttsobject
TTS usage accumulated across all segments of this response.
Show child attributes
modelstring
TTS model used for this response (e.g. inworld-tts-2).
charactersinteger
Total characters synthesized.
audio_secondsnumber
Total assistant audio duration emitted by TTS, in seconds.
sttobject
STT usage for the user audio that fed this response. Drained per response.done from a rolling per-session counter, so each response sees only the audio transcribed since the previous response.done.
Show child attributes
modelstring
STT model used (e.g. inworld/inworld-stt-1).
audio_secondsnumber
User audio duration transcribed for this turn, in seconds.
metadataobject
Metadata echoed from response.create. Omitted when the client did not supply response metadata; never contains LLM Router providerData.metadata.
Example
{
"event_id": "string",
"response": {
"id": "string",
"object": "string",
"status": "in_progress",
"status_details": {
"type": "completed",
"reason": "string",
"error": {
"type": "string",
"code": "string"
}
},
"output": [
{
"object": "string",
"id": "string",
"type": "message",
"status": "completed",
"role": "system",
"content": [
{
"type": "input_text",
"text": "string",
"audio": "string",
"transcript": "string",
"image": "string",
"image_url": "string",
"detail": "low"
}
],
"call_id": "string",
"name": "string",
"arguments": "string",
"output": "string"
}
],
"conversation_id": "string",
"output_modalities": [
"string"
],
"max_output_tokens": 0,
"instructions": "string",
"usage": {
"total_tokens": 0,
"input_tokens": 0,
"output_tokens": 0,
"input_token_details": {
"cached_tokens": 0,
"cache_write_tokens": 0,
"text_tokens": 0,
"audio_tokens": 0
},
"output_token_details": {
"text_tokens": 0,
"audio_tokens": 0,
"reasoning_tokens": 0
},
"llm": {
"model": "string"
},
"tts": {
"model": "string",
"characters": 0,
"audio_seconds": 0
},
"stt": {
"model": "string",
"audio_seconds": 0
}
}
}
}response.output_item.added
An output item was added to the response.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
response_idstringrequired
The response this item belongs to.
output_indexinteger
Index of the output item in the response.
itemobjectrequired
Server conversation item snapshot. Variant-specific function fields are always serialized, including as empty strings.
Show child attributes
objectstring
Object type identifier (read-only, present in server responses).
idstring
Item ID.
typeenum<string>required
Item variant.
Available options:messagefunction_callfunction_call_output
statusenum<string>
Item status (read-only, present in server responses).
Available options:completedin_progressincomplete
roleenum<string>
Available options:systemuserassistanttool
contentobject[]
Show child attributes
typeenum<string>required
Content type. User input uses inputtext/inputaudio/inputimage. Assistant item snapshots use outputtext/outputaudio. Streaming response.contentpart events use text/audio. Legacy text/audio aliases are accepted when creating assistant items.
Available options:input_textinput_audioinput_imagetextaudiooutput_textoutput_audio
textstring
Text content.
audiostring
Base64-encoded audio.
transcriptstring
Human-readable transcript accompanying audio.
imagestring
Inline image data for input_image content.
image_urlstring
Image URL for input_image content.
detailenum<string>
Image detail level.
Available options:lowhighauto
call_idstring
Tool-call identifier. Required and non-empty for client-created functioncalloutput; optional for functioncall. Always present on server functioncall_output snapshots, including as an empty string.
namestring
Function name. Required and non-empty for client-created functioncall; always present on server functioncall snapshots, including as an empty string.
argumentsstring
Function arguments as a JSON string. May be omitted on input and then normalizes to an empty string; always present on server function_call snapshots.
outputstring
Function result. May be omitted on input and then normalizes to an empty string; always present on server functioncalloutput snapshots.
Example
{
"event_id": "string",
"response_id": "string",
"output_index": 0,
"item": {
"object": "string",
"id": "string",
"type": "message",
"status": "completed",
"role": "system",
"content": [
{
"type": "input_text",
"text": "string",
"audio": "string",
"transcript": "string",
"image": "string",
"image_url": "string",
"detail": "low"
}
],
"call_id": "string",
"name": "string",
"arguments": "string",
"output": "string"
}
}response.output_item.done
An output item finished.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
response_idstringrequired
The response this item belongs to.
output_indexinteger
Index of the output item in the response.
itemobjectrequired
Server conversation item snapshot. Variant-specific function fields are always serialized, including as empty strings.
Show child attributes
objectstring
Object type identifier (read-only, present in server responses).
idstring
Item ID.
typeenum<string>required
Item variant.
Available options:messagefunction_callfunction_call_output
statusenum<string>
Item status (read-only, present in server responses).
Available options:completedin_progressincomplete
roleenum<string>
Available options:systemuserassistanttool
contentobject[]
Show child attributes
typeenum<string>required
Content type. User input uses inputtext/inputaudio/inputimage. Assistant item snapshots use outputtext/outputaudio. Streaming response.contentpart events use text/audio. Legacy text/audio aliases are accepted when creating assistant items.
Available options:input_textinput_audioinput_imagetextaudiooutput_textoutput_audio
textstring
Text content.
audiostring
Base64-encoded audio.
transcriptstring
Human-readable transcript accompanying audio.
imagestring
Inline image data for input_image content.
image_urlstring
Image URL for input_image content.
detailenum<string>
Image detail level.
Available options:lowhighauto
call_idstring
Tool-call identifier. Required and non-empty for client-created functioncalloutput; optional for functioncall. Always present on server functioncall_output snapshots, including as an empty string.
namestring
Function name. Required and non-empty for client-created functioncall; always present on server functioncall snapshots, including as an empty string.
argumentsstring
Function arguments as a JSON string. May be omitted on input and then normalizes to an empty string; always present on server function_call snapshots.
outputstring
Function result. May be omitted on input and then normalizes to an empty string; always present on server functioncalloutput snapshots.
Example
{
"event_id": "string",
"response_id": "string",
"output_index": 0,
"item": {
"object": "string",
"id": "string",
"type": "message",
"status": "completed",
"role": "system",
"content": [
{
"type": "input_text",
"text": "string",
"audio": "string",
"transcript": "string",
"image": "string",
"image_url": "string",
"detail": "low"
}
],
"call_id": "string",
"name": "string",
"arguments": "string",
"output": "string"
}
}response.content_part.added
A content part was added to an output item.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
response_idstringrequired
Response identifier.
item_idstring
Item identifier.
output_indexinteger
Index of the output item.
content_indexinteger
Index of the content part.
partobjectrequired
The content part. type: "audio" or "text". transcript: initially empty string.
Example
{
"event_id": "string",
"response_id": "string",
"item_id": "string",
"output_index": 0,
"content_index": 0
}response.content_part.done
A content part finished.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
response_idstringrequired
Response identifier.
item_idstring
Item identifier.
output_indexinteger
Index of the output item.
content_indexinteger
Index of the content part.
partobjectrequired
The completed content part with final transcript.
Example
{
"event_id": "string",
"response_id": "string",
"item_id": "string",
"output_index": 0,
"content_index": 0
}response.output_text.delta
Streaming text chunk from the model.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
response_idstringrequired
Response identifier.
item_idstring
Item identifier.
output_indexinteger
Index of the output item.
content_indexinteger
Index of the content part.
deltastringrequired
Text chunk.
Example
{
"event_id": "string",
"response_id": "string",
"item_id": "string",
"output_index": 0,
"content_index": 0,
"delta": "string"
}response.output_text.done
Text output finished.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
response_idstringrequired
Response identifier.
item_idstring
Item identifier.
output_indexinteger
Index of the output item.
content_indexinteger
Index of the content part.
textstringrequired
Complete text output.
Example
{
"event_id": "string",
"response_id": "string",
"item_id": "string",
"output_index": 0,
"content_index": 0,
"text": "string"
}response.output_audio_transcript.delta
Streaming transcript for generated audio.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
response_idstringrequired
Response identifier.
item_idstring
Item identifier.
output_indexinteger
Index of the output item.
content_indexinteger
Index of the content part.
deltastringrequired
Transcript chunk.
Example
{
"event_id": "string",
"response_id": "string",
"item_id": "string",
"output_index": 0,
"content_index": 0,
"delta": "string"
}response.output_audio_transcript.done
Final transcript for generated audio.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
response_idstringrequired
Response identifier.
item_idstring
Item identifier.
output_indexinteger
Index of the output item.
content_indexinteger
Index of the content part.
transcriptstringrequired
Complete transcript.
Example
{
"event_id": "string",
"response_id": "string",
"item_id": "string",
"output_index": 0,
"content_index": 0,
"transcript": "string"
}response.output_audio.done
Audio output for a content part finished.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
response_idstringrequired
Response identifier.
item_idstring
Item identifier.
output_indexinteger
Index of the output item.
content_indexinteger
Index of the content part.
Example
{
"event_id": "string",
"response_id": "string",
"item_id": "string",
"output_index": 0,
"content_index": 0
}response.function_call_arguments.delta
Streaming function call arguments.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
response_idstringrequired
Response identifier.
item_idstringrequired
Item identifier.
output_indexintegerrequired
Index of the output item.
call_idstringrequired
Tool-call identifier.
deltastringrequired
Arguments chunk (JSON string fragment).
Example
{
"event_id": "string",
"response_id": "string",
"item_id": "string",
"output_index": 0,
"call_id": "string",
"delta": "string"
}response.function_call_arguments.done
Function call arguments finished.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
response_idstringrequired
Response identifier.
item_idstringrequired
Item identifier.
output_indexintegerrequired
Index of the output item.
call_idstringrequired
Tool-call identifier.
namestringrequired
Function name.
argumentsstringrequired
Complete function call arguments (JSON string).
Example
{
"event_id": "string",
"response_id": "string",
"item_id": "string",
"output_index": 0,
"call_id": "string",
"name": "string",
"arguments": "string"
}input_audio_buffer.speech_started
Voice activity detected — user started speaking. Always emitted before transcripts, including on STT providers without native VAD (the server synthesizes the event at first-audio so client code can rely on the same ordering across providers).
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
audio_start_msinteger
Millisecond offset in the audio stream where speech was detected.
item_idstringrequired
The conversation item ID associated with this speech segment.
Example
{
"event_id": "string",
"audio_start_ms": 0,
"item_id": "string"
}input_audio_buffer.speech_stopped
Voice activity ended — user stopped speaking.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
audio_end_msinteger
Millisecond offset in the audio stream where speech ended.
item_idstringrequired
The conversation item ID associated with this speech segment.
Example
{
"event_id": "string",
"audio_end_ms": 0,
"item_id": "string"
}input_audio_buffer.committed
Buffered audio was committed as a conversation item.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
previous_item_idstringnull
The ID of the preceding conversation item, or null.
item_idstringrequired
The new conversation item ID for the committed audio.
Example
{
"event_id": "string",
"item_id": "string"
}input_audio_buffer.cleared
Input audio buffer was cleared.
Payload
typeobjectrequired
event_idstringrequired
Example
{
"event_id": "string"
}input_audio_buffer.timeout_triggered
An idle timeout was triggered on the input buffer. Emitted under servervad when no speech has been detected within `turndetection.idletimeoutms`.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
audio_start_msintegerrequired
Audio buffer start offset (ms) at the time the idle timeout fired.
audio_end_msintegerrequired
Audio buffer end offset (ms) at the time the idle timeout fired.
item_idstringrequired
Conversation item ID associated with the idle audio buffer.
Example
{
"event_id": "string",
"audio_start_ms": 0,
"audio_end_ms": 0,
"item_id": "string"
}input_audio_buffer.turn_suggestion
Emitted by the server VAD smart-turn detector when it predicts the user has reached an end-of-turn boundary. Clients can use this signal to drive low-latency UI cues or to pre-warm a response without waiting for the final speechstopped commit. May be followed by `inputaudiobuffer.turnsuggestion_revoked` if the user resumes speaking.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
item_idstringrequired
The conversation item ID associated with this utterance.
utterance_indexintegerrequired
Monotonic index of the utterance within the session. Pairs with the matching turn_suggestion_revoked.
probabilitynumberrequired
Smart-turn model end-of-turn probability (0.0–1.0).
trailing_silence_msnumber
Trailing silence at the time of inference, in milliseconds.
audio_duration_msnumber
Audio duration of the utterance at the time of inference, in milliseconds.
inference_msnumber
Smart-turn model inference latency, in milliseconds.
Example
{
"event_id": "string",
"item_id": "string",
"utterance_index": 0,
"probability": 0,
"trailing_silence_ms": 0,
"audio_duration_ms": 0,
"inference_ms": 0
}input_audio_buffer.turn_suggestion_revoked
Emitted when the user resumes speaking after a previous turnsuggestion. Pairs with the most recent turnsuggestion sharing the same utterance_index.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
item_idstringrequired
The conversation item ID associated with this utterance.
utterance_indexintegerrequired
Index of the utterance whose previous turn_suggestion is being revoked.
Example
{
"event_id": "string",
"item_id": "string",
"utterance_index": 0
}output_audio_buffer.started
Server started sending output audio.
Payload
typeobjectrequired
event_idstringrequired
Example
{
"event_id": "string"
}output_audio_buffer.stopped
Server stopped sending output audio.
Payload
typeobjectrequired
event_idstringrequired
Example
{
"event_id": "string"
}output_audio_buffer.cleared
Output audio buffer was cleared.
Payload
typeobjectrequired
event_idstringrequired
Example
{
"event_id": "string"
}response.backchannel.audio.delta
Streaming PCM audio chunk for a low-latency back-channel interjection (e.g. "uh-huh", "right") emitted while the user is mid-utterance. Out-of-band from the main response stream — use backchannel_id to group chunks belonging to the same interjection.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
backchannel_idstringrequired
Synthetic ID grouping deltas + done for a single back-channel interjection. Use as the playback bucket key so chunks of one interjection don't collide with the active response item.
deltastringrequired
Base64-encoded audio chunk in the session's configured audio.output.format (PCM16, audio/pcmu, or audio/pcma).
Example
{
"event_id": "string",
"backchannel_id": "string",
"delta": "string"
}response.backchannel.audio.done
All audio for a back-channel interjection has been streamed. No teardown required — playback queues until exhausted.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
backchannel_idstringrequired
Identifies which back-channel interjection finished streaming.
phrasestring
The chosen back-channel utterance (e.g. "uh-huh", "right"). Optional — omitted when the decider doesn't surface the phrase to clients.
Example
{
"event_id": "string",
"backchannel_id": "string",
"phrase": "string"
}response.backchannel.skipped
An evaluation tick chose not to fire a back-channel. Useful for client-side telemetry; clients that don't care can ignore this event.
Payload
typeobjectrequired
event_idstringrequired
Server-generated event ID.
reasonstringrequired
Short machine-readable string describing why no back-channel was emitted on this evaluation tick (e.g. min_gap_not_elapsed, deadline_missed, no_phrase). Stable enough for telemetry; clients that don't care can ignore.
Example
{
"event_id": "string",
"reason": "string"
}rate_limits.updated
Reports current rate limit state.
Payload
typeobjectrequired
event_idstringrequired
Example
{
"event_id": "string"
}