API

Capabilities Configuration

const capabilities = new Capabilities({ emotions: true });

Name	Description	Default value
audio	If `false`, then the client will not receive spoken audio from the characters (text only mode).	`true`
debugInfo	Client wants to receive ControlEvent warning log packets. Disabled when at least one of logs capability is set `true`. Please, use `logsWarning` capability instead.	`false`
emotions	The client will receive character emotions	`false`
interruptions	Allow interruptions to conversations with the character.	`false`
logsDebug	Client wants to receive server debug logs. LogEvent will be received.	`false`
logsInfo	Client wants to receive server info logs. LogEvent will be received.	`true`
logsWarning	Client wants to receive server warning logs. LogEvent will be received.	`true`
phonemes	Include phoneme information in Audio Events.	`false`
silence	Allow for pauses between character replies.	`false`
narratedActions	Allow to receive information about character narrated actions	`false`

Connection Configuration

const connection = {
  disconnectTimeout: 10 * 1000, // time in milliseconds
  autoReconnect: false, // true by default
}

Name	Description	Default value
disconnectTimeout	Close connection due to inactivity	`60000`
autoReconnect	If the current connection is closed, automatically re-open it on send. If this does not occur, an existing open connection is used. The server closes the connection automatically after 1 minute of inactivity.	`true`

SessionToken

const token = client.generateSessionToken();

// Get a token.
token.token

// Get type (Bearer).
token.type

// Get expiration time.
// Token should be regenerated after expiration time.
token.expirationTime

// Get session id.
token.sessionId

Token lifetime duration is 4 Hours by default.

Note: The server implements a 30 Minute session timeout, meaning sessions are expired after 30 Minutes of inactivity is detected. When this occurs, you must recreate the connection to start a new session.

If the token expires while the session is still alive the SDK generates a new token automatically. See Connection configuration for details on default values and how to change them.

InworldClient with API Key

If you prefer to use automatic session management set client key/secret.

const client = new InworldClient();

// ApiKey is required.
client.setApiKey({ key, secret });

// User is not required.
client.setUser(user);

// Configuration is not required.
client.setConfiguration({
  capabilities,
  connection: {
    disconnectTimeout: 10 * 1000, // time in milliseconds
    autoReconnect: false,
  }
});

// Scene is not required for token generation.
// But if you would like to speak with scene characters you need to provide scene full name.
// It should be like workspaces/{WORKSPACE_NAME}/characters/{CHARACTER_NAME}.
// Or like workspaces/{WORKSPACE_NAME}/scenes/{SCENE_NAME}.
client.setScene(scene);

// Event handlers
client.setOnDisconnect(fn);
client.setOnError(fn);
client.setOnWarning(fn); // If no function is specified, console.warn will be invoked.
client.setOnMessage(fn);

// Generate SessionAccessToken.
client.generateSessionToken();

// Finish connection configuration.
// Return instance of EventService.
// Connection is not open. It's just ready to open on message send.
const connection = client.build();

Unitary Session

Starting from version 1.12.0, server-side supports "Unitary Session," enabling several operations to be performed without the need to restart the session.

Change Scene

You can dynamically alter scene names during runtime without the need to recreate connections.

  // You can keep inintial/previous capabilies in application memory or just call a getCapabilities method
  const prevCapabilities = connection.getCapabilities();
  const newCapabilities = { audio: false };

  // Scene name should be like workspaces/{WORKSPACE_NAME}/characters/{CHARACTER_NAME}.
  // Or like workspaces/{WORKSPACE_NAME}/scenes/{SCENE_NAME}.
  await connection.changeScene(sceneName, props?: {
    capablities: { ...prevCapabilities, ...newCapabilities },
  });

Add and Remove Characters

Once a connection is initiated and a Scene Name is set the characters associated with that scene (set in Inworld Studio) become available to talk.

Characters can also be added or removed at runtime.

  const scene = 'workspaces/test-workspace/scenes/test-scene';
  const names = [
    'workspaces/test-workspace/characters/character-1',
    'workspaces/test-workspace/characters/character-2',
    'workspaces/test-workspace/characters/character-3',
  ]

  const client = new InworldClient();

  client.setScene(scene);

  const connection = client.build();

  ...

  await connection.addCharacters(names);

  ...

  await connection.removeCharacters([names[0]]);

Session Management

Sessions expire after 30 Minutes of inactivity. When a session expires conversation history is lost.

The following section details how to preserve previous dialog.

Previous Dialog

You can store all text messages in a location of your choice, allowing you to transfer saved messages to a new session.

For example, consider the following conversation:

Player: 'Hi'
Character: 'Hi, Player.'
Character: 'How can I assist you today?'

You can save this conversation in an array such as this:

const previousDialog = [
  {
    talker: DialogParticipant.PLAYER,
    phrase: 'Hi',
  },
  {
    talker: DialogParticipant.CHARACTER,
    phrase: 'Hi, Player. How can I assist you today?',
  },
]; 

To maintain this conversation after the session has expired, a new connection should be established.

To do this, use the setSessionContinuation method and provide the previousDialog to new connection as shown below:

import { InworldClient, DialogParticipant } from '@inworld/nodejs-sdk';

const client = new InworldClient();

client.setSessionContinuation({ previousDialog });

Previous State

You can retrieve the conversation state from the Inworld AI server using the following method:

import { InworldClient } from '@inworld/nodejs-sdk';

let previousState: string; 
const client = new InworldClient();

client.setOnDisconnect(async () => {
  previousState = (await connection.getSessionState()).state;
});

If you would like to carry on the conversation with the previous state after the session has ended, you need to establish a new connection.

To do this, use the setSessionContinuation method for a new connection and propagate the previousState as shown below:

import { InworldClient } from '@inworld/nodejs-sdk';

const client = new InworldClient();

client.setSessionContinuation({ previousState });

Multi-Characters Conversation

Starting from version 2.6.0, the server-side supports Multi-Character Conversations.

import { InworldClient, InworldTriggers } from '@inworld/nodejs-sdk';

// This scene includes the following characters:
// 'workspaces/test-workspace/characters/character-0'
// 'workspaces/test-workspace/characters/character-1'
// 'workspaces/test-workspace/characters/character-2'
// 'workspaces/test-workspace/characters/character-3'
const scene = 'workspaces/test-workspace/scenes/test-scene';

let conversationId;
const client = new InworldClient();

client.setScene(scene);

const connection = client.build();
const characters = await connection.getCharacters();

// Start conversation with specific characters.
const conversation = connection.startConversation([
  'workspaces/test-workspace/characters/character-0',
  'workspaces/test-workspace/characters/character-1',
]);

// Get conversation id.
conversationId = conversation.getConversationId();

// Send data packets.
await conversation.sendText('Hello');
await conversation.sendTrigger('name', { character: characters[0] });
await conversation.sendNarratedAction('action');
await conversation.sendAudioSessionStart();
await conversation.sendAudioSessionEnd();
await conversation.sendAudio(chunk);
await conversation.sendTTSPlaybackMute(true);
await conversation.sendTTSPlaybackMute(false);

// Force character response.
// It allows characters to speak with each other without user input.
// Just send this trigger each time you want the character to continue the conversation.
await conversation.sendTrigger(InworldTriggers.MUTLI_AGENT_NEXT_TURN);

// Switch to participants that are already in the scene.
await conversation.updateParticipants([
  'workspaces/test-workspace/characters/character-2',
  'workspaces/test-workspace/characters/character-3',
]);

// Or switch to participants that are not in the scene.
// They will be added to the scene automatically.
// These characters should be from the workspace that is used to generate the session token.
// Otherwise, you will get a permission denied error.
await conversation.updateParticipants([
  'workspaces/test-workspace/characters/character-4',
  'workspaces/test-workspace/characters/character-5',
]);

// If an audio session was previously started, it needs to be ended before updating participants.
// Restart the session after completing this call.
await conversation.sendAudioSessionEnd();
await conversation.updateParticipants([
  'workspaces/test-workspace/characters/character-2',
  'workspaces/test-workspace/characters/character-3',
]);
await conversation.sendAudioSessionStart();

InworldConnectionService

const connection = client.build();
// Open connection manually. Available only if configuration.connection.autoReconnect = false.
// Otherwise connection will be managed automatically by SDK.
connection.open();
// You can check if the connection is open or not in case of configuration.connection.autoReconnect = false.
connection.isActive();
// Send text message.
connection.sendText(message: string);
// Send trigger. Pass trigger name and parameters as arguments.
interface TriggerParameter {
  name: string;
  value: string;
}
connection.sendTrigger(name, { parameters: TriggerParameter[] });
// Send narrated action.
connection.sendNarratedAction(text: string);
// Send an audio start event before call sendAudio.
// There are two possible modes defined: MicrophoneMode.EXPECT_AUDIO_END and MicrophoneMode.OPEN_MIC (the default mode).
// There are two possible understandingMode defined: UnderstandingMode.SPEECH_RECOGNITION_ONLY and UnderstandingMode.FULL (the default understandingMode).
connection.sendAudioSessionStart({ mode?: MicrophoneMode, understandingMode?: UnderstandingMode });

Microphone Mode

Mode	Description
`EXPECT_AUDIO_END`	The mic is continuously active and captures audio input without requiring the user to press any button or key.
`OPEN_MIC`	Audio input is only captured when the user presses a button or key. Waits for `sendAudioSessionEnd` call to return final speech recognition result.

Understanding Mode

Mode	Description
`FULL`	The understanding and final results are sent to client as soon as it's ready.
`SPEECH_RECOGNITION_ONLY`	Only final results are sent to client.

// Send audio end event after all audio chunks you would like to send.
connection.sendAudioSessionEnd();
// Send audio. Pass string chunk as argument.
connection.sendAudio(chunk);
// Send a cancel response.
// InteractionId or utteranceId can be omitted.
// When interactionId is empty, everything in the session will be removed.
// When only the utteranceId is provided, nothing happens.
// When only the interactionId is provided, everything until this interaction will be removed.
// When both the interactionId and utteranceId are provided, everything until this interaction will be removed, and utterances in this interaction will also be removed.
connection.sendCancelResponse({
  interactionId?: string,
  utteranceId?: string[],
});
// Close connection.
connection.close();
// Get a character list.
await connection.getCharacters();
// Get the current character.
connection.getCurrentCharacter();
// Change character in the scene.
connection.setCurrentCharacter(character);
// Find character by id.
connection.getCharacterById(id);
// Find character by resource name.
connection.getCharacterByResourceName(resourceName);

User

const user = {
  id: 'user-id',
  fullName: 'FirstName LastName',
  profile: {
    fields: [{ id: 'field_1', value: 'value_1' }]
  },
};

// Globally unique string, id of the end user of the system.
// UUID will be used by default.
user.id
// User name.
user.fullName
// List of user profile fields.
user.profile.fields

Character

const character = connection.getCurrentCharacter();

// Character id.
character.id

// Character resource name.
character.resourceName

// Character display name.
character.displayName

// Character assets.
character.assets.avatarImg
character.assets.avatarImgOriginal
character.assets.rpmModelUri
character.assets.rpmImageUriPortrait
character.assets.rpmImageUriPosture

InworldPacket

client.setOnMessage((packet: InworldPacket) => {...});

// It's a text event.
packet.isText();

// It's an audio event.
packet.isAudio();

// It's a trigger event.
packet.isTrigger();

// It's a task event. It's an incoming event only,
packet.isTask();

// It's a warning. Actual, when debugInfo capability is enabled.
packet.isWarning();

// It's an emotion event.
packet.isEmotion();

// It's a narracted action event.
packet.isNarratedAction();

// It is a log event.  Actual, when logs capabilities are enabled.
packet.isLog();

// It's a control event.
packet.isControl();

// It's a silence event.
packet.isSilence();

// It's a cancel response event.
packet.isCancelResponse();

// It's a special control event. It means that interaction ends.
packet.isInteractionEnd();

// It's a request that SDK sends during scene change.
packet.isSceneMutationRequest();

// It's a request that SDK receives after scene change.
// I.e. when characters added/removed or scene name is changed.
// If only capabilities are changed scene mutation response is not triggered.
packet.isSceneMutationResponse();

client.setOnMessage defines an event listener for incoming messages(packets) with different types of events using the provided methods.

client.setOnMessage((packet: InworldPacket) => {...}); - This line sets up an event listener for the client object. The event listener listens for incoming messages (packets) of type InworldPacket and processes them using the provided callback functions.

Within the callback function, the packet object has several methods to check for specific types of events in the received packet.

Common Event Data

// ISO string.
packet.date

// A token that uniquely identifies the packet.
packet.packetId.packetId
// A token that uniquely identifies and groups utterances in the replies.
// Different packets may belong to the same utterance. E.g. Audio Event and Text Event of the same spoken utterance.
packet.packetId.utteranceId
// A token that uniquely identifies interaction between actors.
packet.packetId.interactionId
// A token that uniquely identifies conversation.
packet.packetId.conversationId

// Determines who sends the packet: player or character.
packet.routing.source.name
packet.routing.source.isPlayer
packet.routing.source.isCharacter

// Determines who receives the packet: player or character.
packet.routing.target.name
packet.routing.target.isPlayer
packet.routing.target.isCharacter

Text Event

// Get message of text event.
packet.text.text
// If this is the final version of the text or not.
// For instance speech recognition may take some time to finalize the text.
packet.text.final

Audio Event

// Get chunk of audio event.
packet.audio.chunk

// Get list if phonemes.
packet.audio.additionalPhonemeInfo = [];

// Get phoneme data.
// Synthesized phoneme.
phonemeInfo.phoneme
// Offset from the beginning of audio segment (in seconds).
phonemeInfo.startOffsetS

Emotion Event

// Get behavior affected by emotions.
packet.emotions.behavior.code.NEUTRAL
packet.emotions.behavior.code.DISGUST
packet.emotions.behavior.code.CONTEMPT
packet.emotions.behavior.code.BELLIGERENCE
packet.emotions.behavior.code.DOMINEERING
packet.emotions.behavior.code.CRITICISM
packet.emotions.behavior.code.ANGER
packet.emotions.behavior.code.TENSION
packet.emotions.behavior.code.TENSE_HUMOR
packet.emotions.behavior.code.DEFENSIVENESS
packet.emotions.behavior.code.WHINING
packet.emotions.behavior.code.SADNESS
packet.emotions.behavior.code.STONEWALLING
packet.emotions.behavior.code.INTEREST
packet.emotions.behavior.code.VALIDATION
packet.emotions.behavior.code.AFFECTION
packet.emotions.behavior.code.HUMOR
packet.emotions.behavior.code.SURPRISE
packet.emotions.behavior.code.JOY

// Get strength of the emotions.
packet.emotions.strength.code.UNSPECIFIED
packet.emotions.strength.code.WEAK
packet.emotions.strength.code.STRONG
packet.emotions.strength.code.NORMAL

Trigger Event

// Trigger name.
packet.trigger.name
// Parameters that come with given event.
packet.trigger.parameters

// Parameter name.
packet.trigger.parameters[i].name
// Parameter value.
packet.trigger.parameter[i].value

Here, [i] is the number associated with the parameter, starting from [0].

Task Event

// Trigger name.
packet.task.name
// Parameters that come with given event.
packet.task.parameters

// Parameter name.
packet.task.parameters[i].name
// Parameter value.
packet.task.parameter[i].value

Here, [i] is the number associated with the parameter, starting from [0].

Narrated Action

// Text.
packet.narratedAction.text

Log Event

// Text.
packet.log.text

// Level: LogLevel.WARNING, LogLevel.INFO, LogLevel.DEBUG.
packet.log.level

// Metadata. Key/value pairs where value is a string.
packet.log.metadata

// Details. JavaScript representation of protobuf value. I.e. it can be string, number, boolean, null, list of protobuf values or key/value pairs where value is a protobuf value.
packet.log.details

Control Event

// Action.
packet.control.action

// Present for warning control event.
packet.control.description

// Actions.
InworlControlAction.INTERACTION_END
InworlControlAction.WARNING

Silence Event

// Silence duration in milliseconds.
packet.silence.durationMs

Cancel Response Event

// Interaction id for response cancellation.
packet.cancelResponses.interaction_id
// Utterance ids to cancel for the given interaction id.
packet.cancelResponses.utterance_id

Interaction End

Interaction End is a type of Control Event that is sent when a character has finished responding to an input.

This event is considered 'true' when:

No interactions have started yet
The last received message is INTERACTION_END

onMessage: (inworldPacket: InworldPacket) => {
  if (inworldPacket.isInteractionEnd()) {
    // Handle end of response
  }
}

Determining which interaction has ended for multiple sent messages can be done by comparing the interactionId, returned when a Player sends a message, to the interactionId received from a responding message event.

const sendPacketA = await connection.sendText('Hi');
const sendPacketB = await connection.sendText('How are you?');

onMessage: (inworldPacket: InworldPacket) => {
  if (inworldPacket.isInteractionEnd()) {
    if (sendPacketA.packetId.interactionId == inworldPacket.packetId.interactionId) {
      // Handle end of response for sent message 'Hi'
    }
    if (sendPacketB.packetId.interactionId == inworldPacket.packetId.interactionId) {
      // Handle end of response for sent message 'How are you?'
    }
  }
}

Scene Mutation Request and Response

// Scene resource name.
packet.sceneMutation.name

// Scene display name.
packet.sceneMutation.displayName

// List of resource names of the characters.
packet.sceneMutation.addedCharacterNames

// List of ids of the characters.
packet.sceneMutation.removedCharacterIds

// List of character objects.
packet.sceneMutation.loadedCharacters

Feedback API

To share your feedback with Inworld, you can use the following API calls:

import {
  DislikeType,
} from '@inworld/nodejs-sdk';


// To send like.
const like = connection.feedback.like({
  interactionId, // Required.
  correlationId, // Optional. Used for response regeneration.
});

// To send dislike.
const dislike = connection.feedback.dislike({
  comment: 'I do not like this response', // Optional.
  interactionId,  // Required.
  correlationId,  // Optional. Used for response regeneration.
  types: [DislikeType.IRRELEVANT], // Optional.
});

// To undo a like or dislike made by mistake.
connection.undo(like.name);
connection.undo(dislike.name);

List of Dislike Types

DislikeType.IRRELEVANT
DislikeType.UNSAFE
DislikeType.UNTRUE
DislikeType.INCORRECT_USE_KNOWLEDGE
DislikeType.UNEXPECTED_ACTION
DislikeType.UNEXPECTED_GOAL_BEHAVIOR
DislikeType.REPETITION
DislikeType.UNSPECIFIED

Error Handling

import {
  status,
  InworldError,
} from '@inworld/nodejs-sdk';

connection.setOnError((err: InworldError) {
  switch (err.code) {
      // Cancelled by server due timeout inactivity.
      case status.ABORTED:
      // Cancelled by client.
      case status.CANCELLED:
          break;
      default:
          console.error(err);
          break;
  }
})

You can read more about possible GRPC status codes here: Status Codes.

// gRPC status code.
err?.code
// Text message.
err.message
// Details array.
err?.details
// The type of error that's being sent to the client
err?.details[0]?.errorType
// How quickly the client should try and retry or reconnect
err?.details[0]?.reconnectType
 // The maximum number of retries to attempt for the client
err?.details[0]?.maxRetries
// Id of the resource.
err?.details[0]?.resourceNotFound?.resourceId
// Details for RESOURCE_NOT_FOUND error type.
err?.details[0]?.resourceNotFound?.resourceType

// Possible error types:
// Session token is expired and needs refresh.
ErrorType.SESSION_TOKEN_EXPIRED
// Session token is completely invalid.
ErrorType.SESSION_TOKEN_INVALID
// Session's resources are temporarily exhausted.
ErrorType.SESSION_RESOURCES_EXHAUSTED
// Billing tokens are exhausted.
ErrorType.BILLING_TOKENS_EXHAUSTED
// Developer account is completely disabled.
ErrorType.ACCOUNT_DISABLED
// Session is invalid due to missing agents or some other reason
ErrorType.SESSION_INVALID
// Resource id is invalid or otherwise could not be found.
ErrorType.RESOURCE_NOT_FOUND
// Safety policies have been violated.
ErrorType.SAFETY_VIOLATION
// The session has timed out due to inactivity.
ErrorType.SESSION_EXPIRED
// The audio session has timed out due to exceeding the maximum duration supported by the Audio Processor. This error can occur during an audio session with mode = EXPECT_AUDIO_END if the maximum duration is surpassed.
ErrorType.AUDIO_SESSION_EXPIRED
// The session has been paused due to inactivity.
ErrorType.SESSION_PAUSED

// Possible reconnection types:
ErrorReconnectionType.UNDEFINED
// Client should not try to reconnect
ErrorReconnectionType.NO_RETRY
// Client can try to reconnect immediately
ErrorReconnectionType.IMMEDIATE
// Client can try to reconnect after given period, specified in reconnectTime
ErrorReconnectionType.TIMEOUT

// Possible resource types:
ErrorResourceType.RESOURCE_TYPE_UNDEFINED
// Conversation
ErrorResourceType.RESOURCE_TYPE_CONVERSATION

Environment

Use these environment variables for local debugging of the package.

Name	Description	Default value
`NODE_SDK_INWORLD_ENGINE_HOST`	API endpoint	`api-engine.inworld.ai:443`
`NODE_SDK_INWORLD_ENGINE_SSL`	Define if Engine connection uses TLS	`true`
`NODE_SDK_INWORLD_LOGGER_LEVEL`	Possible values: debug, warn, error
`NODE_SDK_INWORLD_LOGGER_FILE`	File path for logs

API

Capabilities Configuration​

Connection Configuration​

SessionToken​

InworldClient with API Key​

Unitary Session​

Change Scene​

Add and Remove Characters​

Session Management​

Previous Dialog​

Previous State​

Multi-Characters Conversation​

InworldConnectionService​

Microphone Mode​

Understanding Mode​

User​

Character​

InworldPacket​

Common Event Data​

Text Event​

Audio Event​

Emotion Event​

Trigger Event​

Task Event​

Narrated Action​

Log Event​

Control Event​

Silence Event​

Cancel Response Event​

Interaction End​

Scene Mutation Request and Response​

Feedback API​

List of Dislike Types​

Error Handling​

Environment​