> ## Documentation Index
> Fetch the complete documentation index at: https://docs.inworld.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# WebRTC

> Connect to the Realtime API over WebRTC for low-latency audio, with JSON events over a data channel.

Use WebRTC for low-latency voice through the browser's built-in support. A proxy connects your peer connection to the same realtime service used by [WebSocket](https://docs.inworld.ai/realtime/connect/websocket.md). It converts audio between OPUS and PCM16 and forwards events transparently.

## Endpoint

```
https://api.inworld.ai
```

| Endpoint | Method | Description |
| --- | --- | --- |
| `/v1/realtime/calls` | POST | SDP offer/answer exchange |
| `/v1/realtime/ice-servers` | GET | STUN/TURN server configuration |

## Authentication

Browser clients use a [session token](https://docs.inworld.ai/portal/session-tokens.md) created on your backend. Send it as a Bearer credential. The proxy forwards it to the realtime service.

```
Authorization: Bearer <session-token>
```

Server-side clients can use their complete Base64 API credential:

```text
Authorization: Basic <INWORLD_API_KEY>
```

<Warning>
Keep the API key server-side (see [Security best practices](https://docs.inworld.ai/portal/auth-security.md)). Your backend returns a session token to the browser, never the API key (see examples below).
</Warning>

## Flow

1. Fetch config from your server (session token + ICE servers)
2. Create `RTCPeerConnection` with ICE servers
3. Create data channel `oai-events` + add microphone track
4. Create SDP offer → POST it with optional initial session config to `/v1/realtime/calls` → set SDP answer
5. Data channel opens → send any later `session.update` patches → start conversation

Audio flows via RTP tracks (no manual encode/decode). Events flow via data channel using the same JSON schema as [WebSocket](https://docs.inworld.ai/realtime/connect/websocket.md).

## Session config

You can configure the session during signaling or through the data channel:

- **During signaling:** include the session object in a JSON or multipart `/v1/realtime/calls` request. The proxy sends it upstream as an initial `session.update` before returning the SDP answer. It requires a successful send but does not wait for `session.updated`.
- **After connection:** send `session.update` through the data channel. Partial updates patch the initial configuration.

For example, a JSON signaling request can opt into OpenAI-style tool continuation. `SESSION_TOKEN` is the token returned by your backend:

```javascript
const response = await fetch(`${PROXY}/v1/realtime/calls`, {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${SESSION_TOKEN}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    sdp: offer.sdp,
    session: {
      type: 'realtime',
      providerData: {
        auto_tool_response: false
      }
    }
  })
});

const { sdp } = await response.json();
await pc.setRemoteDescription({ type: 'answer', sdp });
```

You can also send the same session fields later as a WebSocket-compatible `session.update` through the data channel. See [Configuring Models](https://docs.inworld.ai/realtime/usage/using-realtime-models.md) for the full breakdown of STT, LLM, and TTS configuration.

```javascript
dc.send(JSON.stringify({
  type: 'session.update',
  session: {
    type: 'realtime',
    model: 'openai/gpt-4o-mini',
    instructions: 'You are a concise concierge.',
    output_modalities: ['audio', 'text'],
    audio: {
      input: {
        turn_detection: {
          type: 'semantic_vad',
          eagerness: 'medium',
          create_response: true,
          interrupt_response: true
        }
      },
      output: {
        voice: 'Clive',
        model: 'inworld-tts-2',
        speed: 1.0
      }
    },
    tools: [{
      type: 'function',
      name: 'get_weather',
      description: 'Fetch weather for a location',
      parameters: {
        type: 'object',
        properties: { location: { type: 'string' } },
        required: ['location']
      }
    }],
    providerData: {
      stt: {
        voice_profile: true,
        language_hints: ['en-US', 'es-MX'],
        end_of_turn_confidence_threshold: 0.7,
        min_end_of_turn_silence: 200,
        max_turn_silence: 5000,
        vad_threshold: 0.5
      },
      tts: {
        segmenter_strategy: 'sentence',
        steering_handling: 'emit_once',
        language: 'en-US',
        delivery_mode: 'CREATIVE',
        conversational: false
      },
      memory: {
        enabled: true,
        turn_interval: 5,
        max_facts: 50
      },
      backchannel: {
        enabled: true
      },
      responsiveness: {
        enabled: true
      }
    }
  }
}));
```

`providerData` carries Inworld-specific extensions to the OpenAI-compatible session shape — STT tuning, TTS segmentation/steering, automatic memory, and more. Most `providerData` fields are hot-swappable via partial `session.update`, but a few are read only at session open and ignored afterwards (notably `providerData.tts.conversational` and `providerData.tts.user_turn_mode`). See [Inworld Realtime API Extensions](https://docs.inworld.ai/realtime/provider-data.md) for the field-by-field reference.

## Audio

Unlike WebSocket (manual base64 PCM), WebRTC handles audio natively:

- **Input**: browser captures mic and sends OPUS over RTP automatically
- **Output**: proxy sends AI audio back as an RTP track — attach to `<audio>` to play

```javascript
pc.ontrack = (e) => {
  const audio = document.createElement('audio');
  audio.autoplay = true;
  audio.srcObject = new MediaStream([e.track]);
  document.body.appendChild(audio);
};
```

<Note>
`response.output_audio.delta` events are **not** sent through the data channel — audio is delivered via the RTP track instead.
</Note>

## Text & Responses

Same as WebSocket, but sent through the data channel:

```javascript
dc.send(JSON.stringify({
  type: 'conversation.item.create',
  item: { type: 'message', role: 'user', content: [{ type: 'input_text', text: 'Hello!' }] }
}));
dc.send(JSON.stringify({ type: 'response.create' }));
```

## Events

Same event types as [WebSocket](https://docs.inworld.ai/realtime/connect/websocket.md#events), received on the data channel.

## Option 1: Direct WebRTC

Server — serves the page and a `/api/config` endpoint that fetches ICE servers and creates a session token. Save the [server-side mint helper](https://docs.inworld.ai/portal/session-tokens.md#server-side-mint-example) as `mint-session-token.mjs` next to the server file. The API key stays on the server:

<Warning>This is a local-development server, bound to loopback. Before deploying a token endpoint, add your application's authentication, authorization, CSRF protection, and per-user rate limits as described in the [backend token pattern](https://docs.inworld.ai/portal/ephemeral-tokens.md#connect-a-client-through-your-backend).</Warning>

```javascript
import 'dotenv/config';
import { readFileSync } from 'fs';
import { createServer } from 'http';
import { mintSessionToken } from './mint-session-token.mjs';

const html = readFileSync('index.html');
const API_KEY = process.env.INWORLD_API_KEY || '';
const PROXY = 'https://api.inworld.ai';
let port = 3000;

const server = createServer(async (req, res) => {
  const allowedHosts = [`localhost:${port}`, `127.0.0.1:${port}`];
  if (!allowedHosts.includes(req.headers.host) ||
      (req.headers.origin && req.headers.origin !== `http://${req.headers.host}`)) {
    res.writeHead(403).end();
    return;
  }
  if (req.url === '/api/config') {
    res.setHeader('Cache-Control', 'no-store');
    if (req.method !== 'GET') {
      res.writeHead(405, { Allow: 'GET' }).end();
      return;
    }
    let ice = [];
    try {
      const r = await fetch(`${PROXY}/v1/realtime/ice-servers`, {
        headers: { Authorization: `Basic ${API_KEY}` },
      });
      if (r.ok) ice = (await r.json()).ice_servers || [];
    } catch {}
    try {
      const { token, expirationTime } = await mintSessionToken();
      res.writeHead(200, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify({ token, expirationTime, ice_servers: ice, url: `${PROXY}/v1/realtime/calls` }));
    } catch {
      res.writeHead(502, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify({ error: 'Could not create a Realtime session' }));
    }
    return;
  }
  res.writeHead(200, { 'Content-Type': 'text/html' });
  res.end(html);
});
server.on('error', (e) => {
  if (e.code === 'EADDRINUSE') { console.warn(`Port ${port} in use, trying ${++port}…`); server.listen(port, '127.0.0.1'); }
  else throw e;
});
server.listen(port, '127.0.0.1', () => console.log(`Open http://localhost:${port}`));
```

Client — full WebRTC flow in the browser:

```javascript
const configResponse = await fetch('/api/config', { cache: 'no-store' });
if (!configResponse.ok) throw new Error('Could not create a Realtime session');
const cfg = await configResponse.json();
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });

const pc = new RTCPeerConnection({ iceServers: cfg.ice_servers });
const dc = pc.createDataChannel('oai-events', { ordered: true });
stream.getAudioTracks().forEach(t => pc.addTrack(t, stream));

pc.ontrack = (e) => {
  const audio = document.createElement('audio');
  audio.autoplay = true;
  audio.srcObject = new MediaStream([e.track]);
  document.body.appendChild(audio);
};

dc.onopen = () => {
  dc.send(JSON.stringify({
    type: 'session.update',
    session: {
      type: 'realtime',
      model: 'openai/gpt-4o-mini',
      instructions: 'You are a helpful voice assistant.',
      output_modalities: ['audio', 'text'],
      audio: {
        input: { turn_detection: { type: 'semantic_vad', eagerness: 'medium', create_response: true, interrupt_response: true } },
        output: { voice: 'Clive', model: 'inworld-tts-2' }
      }
    }
  }));
};

dc.onmessage = (e) => {
  const msg = JSON.parse(e.data);
  if (msg.type === 'response.output_text.delta') console.log(msg.delta);
  if (msg.type === 'error') console.error(msg.error?.message);
};

const offer = await pc.createOffer();
await pc.setLocalDescription(offer);
// wait for ICE gathering...
const res = await fetch(cfg.url, {
  method: 'POST',
  headers: { 'Content-Type': 'application/sdp', Authorization: `Bearer ${cfg.token}` },
  body: pc.localDescription.sdp,
});
await pc.setRemoteDescription({ type: 'answer', sdp: await res.text() });
```

## Option 2: OpenAI Agents SDK

The [OpenAI Agents SDK](https://github.com/openai/openai-agents-js) manages the full WebRTC lifecycle — peer connection, SDP exchange, mic, and audio playback:

```javascript
import { RealtimeSession, RealtimeAgent, OpenAIRealtimeWebRTC } from '@openai/agents/realtime';

const agent = new RealtimeAgent({
  name: 'assistant',
  instructions: 'You are a helpful voice assistant.',
  model: 'openai/gpt-4o-mini',
});

const configResponse = await fetch('/api/config', { cache: 'no-store' });
if (!configResponse.ok) throw new Error('Could not create a Realtime session');
const cfg = await configResponse.json();
const audioEl = document.createElement('audio');
audioEl.autoplay = true;

const session = new RealtimeSession(agent, {
  transport: new OpenAIRealtimeWebRTC({
    useInsecureApiKey: true,
    audioElement: audioEl,
    changePeerConnection: async (pc) => {
      if (cfg.ice_servers?.length) pc.setConfiguration({ iceServers: cfg.ice_servers });
      return pc;
    },
  }),
  model: 'gpt-4o-realtime-preview-2025-06-03',
});

await session.connect({ url: cfg.url, apiKey: cfg.token });
session.sendMessage('Hello!');
```

The server-side `/api/config` endpoint is identical to Option 1.

## WebSocket vs WebRTC

| | WebSocket | WebRTC |
| --- | --- | --- |
| **Audio** | PCM16 base64 (manual) | OPUS via RTP (native) |
| **Latency** | Higher | Lower (UDP) |
| **NAT** | Not needed | ICE (STUN/TURN) |
| **Events** | WS messages | DataChannel (same schema) |
| **Best for** | Server-side / Node.js | Browser voice apps |

[API reference](https://docs.inworld.ai/api-reference/realtimeAPI/realtime/realtime-websocket.md) for full event schemas.
