Realtime API (WebRTC)

Overview

The Realtime WebRTC API has three parts:

1. Signaling endpoint — Negotiate the WebRTC connection by exchanging an SDP offer and answer.

POST https://api.inworld.ai/v1/realtime/calls

2. ICE servers — STUN/TURN server settings for reliable connections across NATs.

GET https://api.inworld.ai/v1/realtime/ice-servers

3. Data channel events — Once the WebRTC peer connection is established, all Realtime events use the oai-events data channel.

Data Channel: oai-events

Examples

Start with these reference implementations: JavaScript and Python.

Authentication

Browser and mobile clients use a session token created on your backend. Send the token as a Bearer credential:

Authorization: Bearer <session-token>

Server-side clients can use the complete Base64 API credential copied from Portal or the CLI:

Authorization: Basic <INWORLD_API_KEY>

See Authentication for credential formats and header details.

Signaling Endpoints

Create Call

Create a WebRTC call by posting an SDP offer and optional session configuration. If you provide session, the proxy sends it upstream as an initial session.update after the WebSocket connection succeeds and before returning the SDP answer. If that update cannot be sent, call creation fails.

Request body:

Response body:

{
  "sdp": "v=0\r\no=- 4611731400430051336 2 IN IP4 127.0.0.1\r\n...",
  "session": {
    "model": "llama-3.3-70b-versatile",
    "instructions": "You are a helpful assistant.",
    "output_modalities": ["audio", "text"],
    "audio": {
      "input": {
        "transcription": {
          "model": "inworld/inworld-stt-1"
        },
        "turn_detection": {
          "type": "semantic_vad",
          "eagerness": "medium"
        }
      },
      "output": {
        "model": "inworld-tts-2",
        "voice": "Dennis",
        "speed": 1.0
      }
    }
  }
}

{
  "id": "call_abc123",
  "sdp": "v=0\r\no=- 1234567890 2 IN IP4 0.0.0.0\r\n...",
  "ice_servers": [
    {
      "urls": ["stun:stun.l.google.com:19302"]
    },
    {
      "urls": ["turn:turn.example.com:3478"],
      "username": "<TURN_USERNAME>",
      "credential": "<TURN_CREDENTIAL>"
    }
  ]
}

Get ICE Servers

Get STUN and TURN server settings for WebRTC. Use these ICE servers when creating the RTCPeerConnection for reliable connections across NATs and firewalls.

Response body:

{
  "ice_servers": [
    {
      "urls": [
        "stun:stun.l.google.com:19302",
        "stun:stun1.l.google.com:19302"
      ]
    },
    {
      "urls": [
        "turn:34.41.153.85:3478",
        "turn:34.41.153.85:3479?transport=tcp"
      ],
      "username": "1772761055:6e3fa6ea-2ed0-4306-971e-aa5092cb3736",
      "credential": "6eBPxGW2nsktPFzjjbJSF5PK8ow="
    }
  ]
}

Data Channel Events

Once the WebRTC connection is established, send and receive events as JSON messages over the oai-events data channel. It uses the same event protocol as the Realtime WebSocket API.

Client Events

Events sent from the client to the server.

session.update

Update the session configuration. The server responds with a session.updated event.

{
  "type": "session.update",
  "session": {
    "instructions": "You are a friendly voice assistant.",
    "audio": {
      "input": {
        "transcription": {
          "model": "inworld/inworld-stt-1"
        },
        "turn_detection": {
          "type": "semantic_vad",
          "eagerness": "medium",
          "create_response": true,
          "interrupt_response": true
        }
      },
      "output": {
        "model": "inworld-tts-2",
        "voice": "Dennis",
        "speed": 1.0
      }
    }
  }
}

conversation.item.create

Add a conversation item (message, tool result, etc.).

{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "content": [
      { "type": "input_text", "text": "Hello, how are you?" }
    ]
  }
}

conversation.item.truncate

Truncate an assistant message's audio.

conversation.item.delete

Delete a conversation item by ID.

conversation.item.retrieve

Retrieve a conversation item by ID.

response.create

Trigger a model response. The server streams back response events.

{
  "type": "response.create",
  "response": {
    "output_modalities": ["audio", "text"],
    "instructions": "Respond in a cheerful tone."
  }
}

response.cancel

Cancel an in-progress response.

input_audio_buffer.append

Append audio bytes to the input buffer.

input_audio_buffer.commit

Commit the buffered audio as a user message.

input_audio_buffer.clear

Discard all audio in the input buffer.

output_audio_buffer.clear

Clear the server's output audio buffer, stopping playback.

Server Events

Events emitted by the server to the client.

session.created

session.updated

Confirms a session.update was applied.

error

Indicates an error occurred.

conversation.item.added

A new item was added to the conversation.

conversation.item.done

An item finished being populated.

Other conversation item events

conversation.item.input_audio_transcription.delta

Streaming partial transcription for user audio.

conversation.item.input_audio_transcription.completed

Final transcription for a user audio item.

response.created

A new response was created. Contains the full response object in its initial state.

response.done

The response finished. Contains the completed response object with final status and output.

response.output_item.added

An output item was added to the response.

response.output_item.done

An output item finished.

response.content_part.added

A content part was added to an output item.

response.content_part.done

A content part finished.

response.output_text.delta

Streaming text chunk from the model.

response.output_text.done

Text output finished.

response.output_audio_transcript.delta

Streaming transcript for generated audio.

response.output_audio_transcript.done

Final transcript for generated audio.

response.output_audio.delta

Streaming alignment data for generated speech. Over WebRTC, audio travels on the RTP media track. This data-channel event carries only timestamp_info; the delta field is empty. The event is present only when providerData.tts.timestamp_type is set.

response.output_audio.done

Audio output for a content part finished.

response.function_call_arguments.delta

Streaming tool call arguments.

response.function_call_arguments.done

Tool call arguments finished.

input_audio_buffer.speech_started

Voice activity detected — user started speaking.

input_audio_buffer.speech_stopped

Voice activity ended — user stopped speaking.

input_audio_buffer.committed

Buffered audio was committed as a conversation item.

input_audio_buffer.timeout_triggered

The input buffer reached its idle timeout. This event applies only to server VAD and is controlled by turn_detection.idle_timeout_ms.

input_audio_buffer.turn_suggestion

The server VAD smart-turn detector predicts an end-of-turn boundary. Use this signal for low-latency UI cues or to pre-warm a response before the final speech_stopped commit. If the user resumes speaking, input_audio_buffer.turn_suggestion_revoked may follow.

input_audio_buffer.turn_suggestion_revoked

This event is emitted when the user resumes speaking after a turn_suggestion. It pairs with the most recent turn_suggestion that has the same utterance_index.

Other audio buffer events

response.backchannel.audio.delta

A streaming PCM audio chunk for a low-latency back-channel interjection, such as "uh-huh" or "right", while the user is speaking. It is separate from the main response stream. Use backchannel_id to group chunks from the same interjection.

response.backchannel.audio.done

All audio for a back-channel interjection has been streamed. No teardown is required. Queued audio plays until the queue is empty.

response.backchannel.skipped

A back-channel evaluation chose not to emit an interjection. Use this event for client-side telemetry, or ignore it if you do not need that information.

rate_limits.updated

Reports current rate limit state.

Schemas

Session object

The session object configures model behavior, audio settings, tools, and more. It appears in signaling requests, session.update, session.created, and session.updated events.

AudioConfig

AudioFormat

TurnDetection

Turn detection has two modes, selected by the type field.

Server VAD (type: "server_vad"):

Semantic VAD (type: "semantic_vad"):

ConversationItem

ContentPart

ResponseConfig

Per-response overrides for session defaults.

Tool

ToolChoiceTarget

Specifies tool choice behavior. The server always returns this as an object.