Guides

Managing Conversations

Learn how to manage conversation history and context with the Realtime API

Conversation Items

Conversation items represent messages and interactions in your conversation. Each item has:

ID: Unique identifier
Type: message, function_call, function_call_output
Role: user, assistant, or tool
Content: The actual content of the item (array of content parts)

Content Types

Conversation items support different content types depending on direction:

Input Content Types (for user messages):

input_text - Plain text input from the user
input_audio - Base64-encoded audio input from the user
input_image - Image input with a non-empty image or image_url and optional detail (low, high, or auto)

Output content types depend on the event:

Assistant item snapshots in conversation.item.*, response.output_item.*, and response.done use output_text and output_audio.
Streaming response.content_part.* events use text and audio.

When creating an assistant item, use the canonical output_text and output_audio spellings. The legacy text and audio aliases are also accepted and normalized.

Creating Conversation Items

Text Messages

javascript

ws.send(JSON.stringify({
  type: 'conversation.item.create',
  item: {
    type: 'message',
    role: 'user',
    content: [
      {
        type: 'input_text',
        text: 'Hello, how are you?'
      }
    ]
  }
}));

Positioning items

Use previous_item_id on conversation.item.create to control where the new item is inserted:

Value	Behavior
Omitted	Append after the latest item.
`"root"`	Insert at the beginning of the conversation.
Existing item ID	Insert immediately after that item.
Unknown item ID	Reject with `item_not_found` and `error.param: "previous_item_id"`; history is unchanged.

javascript

ws.send(JSON.stringify({
  type: 'conversation.item.create',
  previous_item_id: 'item_123',
  item: {
    type: 'message',
    role: 'user',
    content: [{ type: 'input_text', text: 'Insert this after item_123.' }]
  }
}));

The server reports the actual predecessor in conversation.item.added and conversation.item.done. The first item has previous_item_id: null.

Audio Messages

There are two ways to send audio input:

Method 1: Streaming Audio (Real-time) Use input_audio_buffer.append for streaming real-time audio from a microphone:

javascript

// Stream audio chunks in real-time
ws.send(JSON.stringify({
  type: 'input_audio_buffer.append',
  audio: base64AudioData
}));
// VAD automatically detects speech boundaries and commits the buffer

Method 2: Pre-recorded Audio Chunks Use conversation.item.create with input_audio for pre-recorded audio chunks:

javascript

ws.send(JSON.stringify({
  type: 'conversation.item.create',
  item: {
    type: 'message',
    role: 'user',
    content: [{
      type: 'input_audio',
      audio: base64AudioData  // Base64-encoded PCM16 or OPUS audio
    }]
  }
}));

When to use each method:

Streaming (input_audio_buffer.append): Use for real-time microphone input, voice conversations, live audio streaming
Pre-recorded (conversation.item.create with input_audio): Use for pre-recorded audio files, batch processing, or when you have complete audio chunks ready

Mixed Content

You can combine multiple content types in a single conversation item:

javascript

ws.send(JSON.stringify({
  type: 'conversation.item.create',
  item: {
    type: 'message',
    role: 'user',
    content: [
      {
        type: 'input_text',
        text: 'Here is some context about the audio:'
      },
      {
        type: 'input_audio',
        audio: base64AudioData
      },
      {
        type: 'input_text',
        text: 'And here is additional context.'
      }
    ]
  }
}));

Receiving Conversation Items

When items are added to the conversation, you'll receive events:

javascript

ws.on('message', (data) => {
  const event = JSON.parse(data);
  
  if (event.type === 'conversation.item.added') {
    console.log('Item added:', event.item.id);
    console.log('Content:', event.item.content);
  }
  
  if (event.type === 'conversation.item.done') {
    console.log('Item processing complete:', event.item.id);
  }
});

Retrieving Conversation Items

Retrieve specific conversation items:

javascript

ws.send(JSON.stringify({
  type: 'conversation.item.retrieve',
  item_id: 'item-id-here'
}));

The server will respond with:

javascript

{
  type: 'conversation.item.retrieved',
  item: {
    id: 'item-id-here',
    type: 'message',
    role: 'user',
    content: [...]
  }
}

Deleting Conversation Items

Remove items from the conversation:

javascript

ws.send(JSON.stringify({
  type: 'conversation.item.delete',
  item_id: 'item-id-here'
}));

You'll receive a confirmation:

javascript

{
  type: 'conversation.item.deleted',
  item_id: 'item-id-here'
}

Function Calling

The Realtime API supports function calling, allowing the assistant to invoke tools you define. Configure functions in session.update and handle function call events.

Defining Functions

javascript

ws.send(JSON.stringify({
  type: 'session.update',
  session: {
    type: 'realtime',
    tools: [{
      type: 'function',
      name: 'get_weather',
      description: 'Get the weather for a location',
      parameters: {
        type: 'object',
        properties: {
          location: {
            type: 'string',
            description: 'The city and state, e.g. San Francisco, CA'
          }
        },
        required: ['location']
      }
    }],
    tool_choice: 'auto'
  }
}));

Handling Function Calls

javascript

ws.on('message', (data) => {
  const event = JSON.parse(data);
  
  if (event.type === 'response.function_call_arguments.done') {
    const result = executeFunction(event.name, JSON.parse(event.arguments));
    
    ws.send(JSON.stringify({
      type: 'conversation.item.create',
      item: {
        type: 'function_call_output',
        call_id: event.call_id,
        output: JSON.stringify(result)
      }
    }));
    
  }
});

By default, providerData.auto_tool_response is true, so adding the function_call_output automatically starts the follow-up response. For OpenAI-style client-controlled continuation, set providerData.auto_tool_response to false in session.update, then send response.create after the output item.

Voice Activity Detection

Voice Activity Detection (VAD) automatically detects when speech starts and stops, enabling natural turn-taking in conversations. Configure VAD through session.update.

Configuring VAD

javascript

ws.send(JSON.stringify({
  type: 'session.update',
  session: {
    type: 'realtime',
    audio: {
      input: {
        turn_detection: {
          type: 'semantic_vad',
          eagerness: 'medium',
          create_response: true,
          interrupt_response: true
        }
      }
    }
  }
}));

VAD Types

semantic_vad: Uses conversational awareness to detect natural speech boundaries. Adjust eagerness (low, medium, high) to control responsiveness.

VAD Events

javascript

ws.on('message', (data) => {
  const event = JSON.parse(data);
  
  if (event.type === 'input_audio_buffer.speech_started') {
    console.log('Speech detected');
    // Update UI to show user is speaking
  }
  
  if (event.type === 'input_audio_buffer.speech_stopped') {
    console.log('Speech ended');
    // Update UI, prepare for response
  }
});

Error Handling

The Realtime API emits error events for various failure scenarios. Handle these events to provide robust error recovery and user feedback.

Error Event Structure

javascript

ws.on('message', (data) => {
  const event = JSON.parse(data);
  
  if (event.type === 'error') {
    const error = event.error;
    
    switch (error.type) {
      case 'invalid_request_error':
        console.error('Invalid request:', error.message);
        if (error.param) {
          console.error('Parameter:', error.param);
        }
        break;
      case 'server_error':
        console.error('Server error:', error.message);
        // Implement retry logic
        break;
      case 'rate_limit_error':
        console.error('Rate limit exceeded');
        // Pause requests, implement backoff
        break;
    }
  }
});

Error Types

invalid_request_error: Invalid parameters or malformed requests. Check error.param for the specific field.
server_error: Transient server-side failures. Implement retry logic with exponential backoff.
rate_limit_error: Rate limit exceeded. Throttle requests and retry with exponential backoff. See Handling rate-limited requests for implementation details.

Interruption Handling

Interrupt active responses when new user input arrives.

Interrupting Responses

Cancel an in-progress response when the user starts speaking again:

javascript

ws.on('message', (data) => {
  const event = JSON.parse(data);
  
  if (event.type === 'input_audio_buffer.speech_started') {
    // User started speaking, cancel current response
    ws.send(JSON.stringify({
      type: 'response.cancel'
    }));
  }
});

When interrupt_response: true is set in VAD configuration, the server automatically cancels responses when new speech is detected.

Managing Context

Session Instructions

Update session instructions to guide the conversation:

javascript

ws.send(JSON.stringify({
  type: 'session.update',
  session: {
    type: 'realtime',
    instructions: 'You are a helpful assistant. Be concise and friendly.'
  }
}));

Conversation History

The API automatically maintains conversation history. You can:

Keep full history: Let the conversation grow naturally
Selective deletion: Remove specific items that aren't needed
Session resets: Start a new session when you need a clean context window

Example: Conversation Manager

Here's a complete example of managing conversations:

javascript

class ConversationManager {
  constructor(ws) {
    this.ws = ws;
    this.items = new Map();
    this.setupListeners();
  }
  
  setupListeners() {
    this.ws.on('message', (data) => {
      const event = JSON.parse(data);
      
      switch (event.type) {
        case 'conversation.item.added':
          this.items.set(event.item.id, event.item);
          break;
        case 'conversation.item.deleted':
          this.items.delete(event.item_id);
          break;
      }
    });
  }
  
  sendMessage(text) {
    this.ws.send(JSON.stringify({
      type: 'conversation.item.create',
      item: {
        type: 'message',
        role: 'user',
        content: [{
          type: 'input_text',
          text: text
        }]
      }
    }));
  }
  
  deleteItem(itemId) {
    this.ws.send(JSON.stringify({
      type: 'conversation.item.delete',
      item_id: itemId
    }));
  }
  
  getConversationHistory() {
    return Array.from(this.items.values());
  }
}

Best Practices

Monitor Context Length: Keep track of conversation length to avoid exceeding limits
Strategic Deletion: Remove old context that's no longer relevant
Item Tracking: Maintain a local map of conversation items for quick access
Error Handling: Handle cases where items might not exist when deleting/retrieving
Context Management: Use session instructions to guide conversation behavior

Use Cases

Long Conversations: Delete old context to maintain performance
Error Recovery: Delete incorrect items and resend
Context Switching: Clear conversation context when changing topics
Memory Management: Remove items that are no longer needed