API

Capabilities configuration

const capabilities = new Capabilities({ emotions: true });

Name	Description	Default value
audio	If false, then the client will not receive spoken audio from the characters (text only mode).	true
debugInfo	Client wants to receive ControlEvent warning log packets. Disabled when capability logs is set true.	false
emotions	The client will receive character emotions.	false
interruptions	Allow interruptions to conversations with the character.	false
logs	Client receives all logs using the LogEvent packet. If false, only warning logs are received using the ControlEvent packet. Requires the capability debugInfo to be true.	false
logsDebug	Client wants to receive server debug logs. Requires: That the capability logs to be true.	false
logsInfo	Client wants to receive server info logs. Requires: That the capability logs to be true.	true
logsWarning	Client wants to receive server warning logs. Requires: That the capability logs to be true.	true
phonemes	Include phoneme information in Audio Events.	false
silence	Allow for pauses between character replies.	false
narratedActions	Allow to receive information about character narrated actions	false

Connection configuration

const connection = {
  disconnectTimeout: 10 * 1000, // time in milliseconds
  autoReconnect: false, // true by default
}

Name	Description	Default value
disconnectTimeout	Close connection due to inactivity	60000
autoReconnect	Connection will be opened automatically on send if closed, or else an existing open connection will be used. Our server closes the connection automatically after 1 minute of inactivity.	true

SessionToken

It is unsafe to use an API KEY directly on the client side so it is better to have a server generate the token in a safe way. The source code below can be used for this purpose.

async function generateSessionToken() {
  const response = await fetch(config.GENERATE_TOKEN_URL);

  return response.json();
}

InworldClient

const client = new InworldClient();

// User is not required.
client.setUser(user);

// Configuration is not required.
client.setConfiguration({
  capabilities,
  connection: {
    autoReconnect: false,
    audioPlayback: {
      // Adjust the volume setting from 1 to 0 when muting, stopping, or experiencing interruptions.
      stop: {
        // Duration for audio playback cessation, measured in milliseconds.
        // The default value is 500 milliseconds.
        duration: 500,
        // Count of ticks.
        // Default value is 25.
        ticks: 25,
      },
    },
    // Time in millisecond.
    disconnectTimeout: 10 * 1000,
    // Specify host for local debugging of the package.
    gateway: {
      hostname: 'hostname:port',
      ssl: true,
    },
  }
});

// An API key is required to generate a session token.
// It is unsafe to use this key directly on the client side so it is better to have server that can generate the token in a safe way.
// Use Node.js SDK to generate the session token.
// You need to fetch the token using the Node.js service.
client.setGenerateSessionToken(fn);

// It should be like workspaces/{WORKSPACE_NAME}/characters/{CHARACTER_NAME}.
// Or like workspaces/{WORKSPACE_NAME}/scenes/{SCENE_NAME}.
client.setScene(scene);

// Main event handlers
client.setOnDisconnect(fn);
client.setOnError(fn);
client.setOnWarning(fn); // If no function is specified, console.warn will be invoked.
client.setOnMessage(fn);
client.setOnReady(fn);

// History change handler.
// History contains all incoming and outgoing events.
// Just display the items on the web page one by one.
client.setOnHistoryChange((history: HistoryItem[]) => {
  console.log(history);
});

// Audio event handlers.
client.setOnBeforePlaying((packet: InworldPacket) => {
  // Do something with the packet before playing.
});
client.setOnAfterPlaying((packet: InworldPacket) => {
  // Do something with the packet after playing.
});
client.setOnStopPlaying(() => {
  // Do something when all interaction packages are played.
});

// Finish connection configuration.
// Return instance of EventService.
// The connection is not yet open but will open on message send.
const connection = client.build();

InworldConnectionService

const connection = client.build();

// Open the connection manually. This is available only if configuration.connection.autoReconnect = false.
// Otherwise the connection will be managed automatically by the SDK.
connection.open();

// You can check if the connection is open or not in the case of configuration.connection.autoReconnect = false.
connection.isActive();

// Send a message.
connection.sendText(message);

// Send trigger. Pass trigger name and parameters as arguments.
interface TriggerParameter {
  name: string;
  value: string;
}
connection.sendTrigger(name, { parameters: TriggerParameter[] });

// Send narrated action.
connection.sendNarratedAction(text: string);

// Send an audio start event before call sendAudio.
// There are two possible modes defined: MicrophoneMode.EXPECT_AUDIO_END and MicrophoneMode.OPEN_MIC (the default mode).
connection.sendAudioSessionStart({ mode?: MicrophoneMode });

Mode	Description
EXPECT_AUDIO_END	The mic is continuously active and captures audio input without requiring the user to press any button or key.
OPEN_MIC	Audio input is only captured when the user presses a button or key. Waits for sendAudioSessionEnd call to return final speech recognition result

// Send an audio end event after all audio chunks that you would like to send.
connection.sendAudioSessionEnd();

// Send an audio. Pass string chunk as argument.
connection.sendAudio(chunk);

// Send a cancel response.
// InteractionId or utteranceId can be omitted.
// When interactionId is empty, everything in the session will be removed.
// When only the utteranceId is provided, nothing happens.
// When only the interactionId is provided, everything until this interaction will be removed.
// When both the interactionId and utteranceId are provided, everything until this interaction will be removed, and utterances in this interaction will also be removed.
connection.sendCancelResponse({
  interactionId?: string,
  utteranceId?: string[],
});

// Close the connection.
connection.close();

// Get a character list.
connection.getCharacters();

// Get the current character.
connection.getCurrentCharacter();

// Change the character in the scene.
connection.setCurrentCharacter(character);

User

const user = {
  id: 'user-id',
  fullName: 'FirstName LastName',
  profile: {
    fields: [{ id: 'field_1', value: 'value_1' }]
  },
};

// Globally unique string, id of the end user of the system.
// UUID will be used by default.
user.id
// User name.
user.fullName
// List of user profile fields.
user.profile.fields

Character

const character = connection.getCurrentCharacter();

// Character id.
character.id

// Character resource name.
character.resourceName

// Character display name.
character.displayName

// Character assets.
character.assets.avatarImg
character.assets.avatarImgOriginal
character.assets.rpmModelUri
character.assets.rpmImageUriPortrait
character.assets.rpmImageUriPosture

InworldPacket

client.setOnMessage((packet: InworldPacket) => {...});

// It is a text event.
packet.isText();

// It is an audio event.
packet.isAudio();

// It is a trigger event.
packet.isTrigger();

// It is an emotion event.
packet.isEmotion();

// It is a log event.
packet.isLog();

// It is a silence event.
packet.isSilence();

// It's a narrated action event.
packet.isNarratedAction();

// It is a control event.
packet.isControl();

// It is a special control event, indicating that interaction has ended.
packet.isInteractionEnd();

// It is a mute control event.
packet.isTTSPlaybackMute();

// It is a unmute control event.
packet.isTTSPlaybackUnmute();

Common Event Data

// ISO string.
packet.date


// A token that uniquely identifies the packet.
packet.packetId.packetId
// A token that uniquely identifies and groups utterances in the replies.
// Different packets may belong to the same utterance. E.g. Audio Event and Text Event of the same spoken utterance.
packet.packetId.utteranceId
// A token that uniquely identifies interaction between actors.
packet.packetId.interactionId
// A token that uniquely identifies a conversation.
packet.packetId.conversationId

// Determines who sends the packet: player or character.
packet.routing.source.name
packet.routing.source.isPlayer
packet.routing.source.isCharacter

// Determines who receives the packet: player or character.
packet.routing.target.name
packet.routing.target.isPlayer
packet.routing.target.isCharacter

Text Event

// Get message of text event.
packet.text.text
// If this is the final version of the text or not.
// For instance speech recognition may take some time to finalize the text.
packet.text.final

Audio Event

// Get chunk of audio event.
packet.audio.chunk
// Get chunk duration in milliseconds.
// Provided by Web API.
// Available only when metadata is loaded in onBeforePlaying and onAfterPlaying callbacks.
// See setOnBeforePlaying and setOnAfterPlaying handlers.
packet.audio.durationMs

// Get list if phonemes.
packet.audio.additionalPhonemeInfo = [];

// Get phoneme data.
// Synthesized phoneme.
phonemeInfo.phoneme
// Offset from the beginning of audio segment (in seconds).
phonemeInfo.startOffsetS

Emotion Event

// Get behavior affected by emotions.
packet.emotion.behavior.code.NEUTRAL
packet.emotion.behavior.code.DISGUST
packet.emotion.behavior.code.CONTEMPT
packet.emotion.behavior.code.BELLIGERENCE
packet.emotion.behavior.code.DOMINEERING
packet.emotion.behavior.code.CRITICISM
packet.emotion.behavior.code.ANGER
packet.emotion.behavior.code.TENSION
packet.emotion.behavior.code.TENSE_HUMOR
packet.emotion.behavior.code.DEFENSIVENESS
packet.emotion.behavior.code.WHINING
packet.emotion.behavior.code.SADNESS
packet.emotion.behavior.code.STONEWALLING
packet.emotion.behavior.code.INTEREST
packet.emotion.behavior.code.VALIDATION
packet.emotion.behavior.code.AFFECTION
packet.emotion.behavior.code.HUMOR
packet.emotion.behavior.code.SURPRISE
packet.emotion.behavior.code.JOY

// Get strength of the emotion.
packet.emotion.strength.code.UNSPECIFIED
packet.emotion.strength.code.WEAK
packet.emotion.strength.code.STRONG
packet.emotion.strength.code.NORMAL

Trigger Event

// Trigger name.
packet.trigger.name
// Parameters that come with given event.
packet.parameters

// Parameter name.
parameter.name
// Parameter value.
parameter.value

Control Event

packet.control.type.INTERACTION_END
packet.control.type.TTS_PLAYBACK_MUTE
packet.control.type.TTS_PLAYBACK_UNMUTE
packet.control.type.WARNING

Silence Event

// Silence duration in milliseconds.
packet.silence.durationMs

Narrated Action Event

packet.narratedAction.text

Cancel Response Event

// Interaction id for response cancellation.
packet.cancelResponses.interaction_id
// Utterance ids to cancel for the given interaction id.
packet.cancelResponses.utterance_id

Interaction End

Interaction End is a type of Control Event to determine when the Character has ended responding to a sent message. It is considered ended when the chat has not started ( there is no actor action yet ) or the last received message is INTERACTION_END.

onMessage: (inworldPacket: InworldPacket) => {
  if (inworldPacket.isInteractionEnd()) {
    // Handle end of response
  }
}

Determining which interaction has ended for multiple sent messages can be done by comparing the interactionId, returned when a Player sends a message, to the interactionId received from a responding message event.

const sendPacketA = await connection.sendText('Hi');
const sendPacketB = await connection.sendText('How are you?');

onMessage: (inworldPacket: InworldPacket) => {
  if (inworldPacket.isInteractionEnd()) {
    if (sendPacketA.packetId.interactionId == inworldPacket.packetId.interactionId) {
      // Handle end of response for sent message 'Hi'
    }
    if (sendPacketB.packetId.interactionId == inworldPacket.packetId.interactionId) {
      // Handle end of response for sent message 'How are you?'
    }
  }
}

Feedback API

To share your feedback with us, you can utilize the following API calls:

import {
  DislikeType,
} from '@inworld/web-core';


// To send like.
const like = connection.feedback.like({
  interactionId, // Required.
  correlationId, // Optional. Used for response regeneration.
});

// To send dislike.
const dislike = connection.feedback.dislike({
  comment: 'I do not like this response because ...', // Optional.
  interactionId,  // Required.
  correlationId,  // Optional. Used for response regeneration.
  types: [DislikeType.IRRELEVANT], // Optional.
});

// To undo a like or dislike made by mistake.
connection.undo(like.name);
connection.undo(dislike.name);

List of Dislike Types

DislikeType.IRRELEVANT
DislikeType.UNSAFE
DislikeType.UNTRUE
DislikeType.INCORRECT_USE_KNOWLEDGE
DislikeType.UNEXPECTED_ACTION
DislikeType.UNEXPECTED_GOAL_BEHAVIOR
DislikeType.REPETITION
DislikeType.UNSPECIFIED

Unitary Session

Starting from version 2.5.0, our server-side supports "Unitary Session," enabling us to perform several operations without the need to restart the session.

Reload or change scene

You can dynamically alter scene names during runtime without the need to recreate connections.

  // Scene name should be like workspaces/{WORKSPACE_NAME}/characters/{CHARACTER_NAME}.
  // Or like workspaces/{WORKSPACE_NAME}/scenes/{SCENE_NAME}.
  await connection.changeScene(sceneName);

Add and remove characters

Once you initiate the connection and designate the scene name, only characters associated with that scene are accessible for conversation. However, you have the flexibility to introduce additional character(s) into the conversation without having to include them in the scene via the Studio UI interface. Additionally, characters can be removed from the scene during runtime.

  const scene = 'workspaces/test-workspace/scenes/test-scene';
  const names = [
    'workspaces/test-workspace/characters/character-1',
    'workspaces/test-workspace/characters/character-2',
    'workspaces/test-workspace/characters/character-3',
  ]

  const client = new InworldClient();

  client.setScene(scene);

  const connection = client.build();

  ...

  await connection.addCharacters(names);

  ...

  await connection.removeCharacters([names[0]]);

Session management

Session is expired after 30 minutes of inactivity, resulting in the loss of previous conversation history. Nonetheless, there's a solution to preserve this history.

Previous dialog

You can store all text messages in a location of your preference, allowing you to transfer saved messages to a new session.

For instance, consider the following conversation:

Player: 'Hi'
Character: 'Hi, Player.'
Character: 'How can I assist you today?'

You can save it in an array like this:

const previousDialog = [
  {
    talker: DialogParticipant.PLAYER,
    phrase: 'Hi',
  },
  {
    talker: DialogParticipant.CHARACTER,
    phrase: 'Hi, Player. How can I assist you today?',
  },
]; 

To maintain this conversation after the session has expired, you should establish a new connection. Simply use the setSessionContinuation method and provide the previousDialog to new connection as shown below:

import { InworldClient, DialogParticipant } from '@inworld/web-core';

const client = new InworldClient();

client.setSessionContinuation({ previousDialog });

Previous state

You can retrieve the conversation state from the Inworld AI server using the following method:

import { InworldClient } from '@inworld/web-core';

let previousState: string; 
const client = new InworldClient();

client.setOnDisconnect(async () => {
  previousState = (await connection.getSessionState()).state;
});

If you'd like to carry on the conversation with the previous state after the session has ended, you'll need to establish a new connection. Simply use the setSessionContinuation method for new connection and propagate the previousState as demonstrated below:

import { InworldClient } from '@inworld/web-core';

const client = new InworldClient();

client.setSessionContinuation({ previousState });

Multi-characters conversation

Starting from version 2.6.0, our server-side supports "Multi-characters conversations".

import { InworldClient, InworldTriggers } from '@inworld/web-core';

// This scene includes the following characters:
// 'workspaces/test-workspace/characters/character-0'
// 'workspaces/test-workspace/characters/character-1'
// 'workspaces/test-workspace/characters/character-2'
// 'workspaces/test-workspace/characters/character-3'
const scene = 'workspaces/test-workspace/scenes/test-scene';

let conversationId;
const client = new InworldClient();

client.setScene(scene);
client.setOnHistoryChange(history => {
  // If you create several conversations per connection, filter history by conversationId.
  // If you use only one conversation, you don't need to filter history items.
  const conversationHistory = history.filter((h) => h.conversationId === conversationId);
});

const connection = client.build();
const characters = await connection.getCharacters();

// Start conversation with specific characters.
const conversation = connection.startConversation([
  'workspaces/test-workspace/characters/character-0',
  'workspaces/test-workspace/characters/character-1',
]);

// Get conversation id.
conversationId = conversation.getConversationId();

// Send data packets.
await conversation.sendText('Hello');
await conversation.sendTrigger('name', { character: characters[0] });
await conversation.sendNarratedAction('action');
await conversation.sendAudioSessionStart();
await conversation.sendAudioSessionEnd();
await conversation.sendTTSPlaybackMute(true);
await conversation.sendTTSPlaybackMute(false);

// Force character response.
// It allows characters to speak with each other without user input.
// Just send this trigger each time you want the character to continue the conversation.
await conversation.sendTrigger(InworldTriggers.MUTLI_AGENT_NEXT_TURN);

// Get conversation information.
conversation.getTranscript();
conversation.getCharacters();
conversation.getHistory();

// Switch to participants that are already in the scene.
await conversation.updateParticipants([
  'workspaces/test-workspace/characters/character-2',
  'workspaces/test-workspace/characters/character-3',
]);

// Or switch to participants that are not in the scene.
// They will be added to the scene automatically.
// These characters should be from the workspace that is used to generate the session token.
// Otherwise, you will get a permission denied error.
await conversation.updateParticipants([
  'workspaces/test-workspace/characters/character-4',
  'workspaces/test-workspace/characters/character-5',
]);

Error handling

import {
  InworldError,
} from '@inworld/nodejs-sdk';

connection.setOnError((err: InworldError) {
  switch (err.code) {
      // Cancelled by server due timeout inactivity (ABORTED).
      case '10':
          break;
      default:
          console.error(err);
          break;
  }
})

You can read more about possible GRPC status codes here.

// gRPC status code.
err?.code
// Text message.
err.message
// Details array.
err?.details
// The type of error that's being sent to the client
err?.details[0]?.errorType
// How quickly the client should try and retry or reconnect
err?.details[0]?.reconnectType
 // The maximum number of retries to attempt for the client
err?.details[0]?.maxRetries
// Id of the resource.
err?.details[0]?.resourceNotFound?.resourceId
// Details for RESOURCE_NOT_FOUND error type.
err?.details[0]?.resourceNotFound?.resourceType

// Possible error types:
// Session token is expired and needs refresh.
ErrorType.SESSION_TOKEN_EXPIRED
// Session token is completely invalid.
ErrorType.SESSION_TOKEN_INVALID
// Session's resources are temporarily exhausted.
ErrorType.SESSION_RESOURCES_EXHAUSTED
// Billing tokens are exhausted.
ErrorType.BILLING_TOKENS_EXHAUSTED
// Developer account is completely disabled.
ErrorType.ACCOUNT_DISABLED
// Session is invalid due to missing agents or some other reason
ErrorType.SESSION_INVALID
// Resource id is invalid or otherwise could not be found.
ErrorType.RESOURCE_NOT_FOUND
// Safety policies have been violated.
ErrorType.SAFETY_VIOLATION
// The session has timed out due to inactivity.
ErrorType.SESSION_EXPIRED
// The audio session has timed out due to exceeding the maximum duration supported by the Audio Processor. This error can occur during an audio session with mode = EXPECT_AUDIO_END if the maximum duration is surpassed.
ErrorType.AUDIO_SESSION_EXPIRED
// The session has been paused due to inactivity.
ErrorType.SESSION_PAUSED

// Possible reconnection types:
ErrorReconnectionType.UNDEFINED
// Client should not try to reconnect
ErrorReconnectionType.NO_RETRY
// Client can try to reconnect immediately
ErrorReconnectionType.IMMEDIATE
// Client can try to reconnect after given period, specified in reconnectTime
ErrorReconnectionType.TIMEOUT

// Possible resource types:
ErrorResourceType.RESOURCE_TYPE_UNDEFINED
// Conversation
ErrorResourceType.RESOURCE_TYPE_CONVERSATION

Known Issues

• To avoid any potential issues with browser audio autoplay support you should send audio packets only after the user has initiated some kind of action on UI (e.g., pressing a button).