API
Capabilities configuration
const capabilities = new Capabilities({ emotions: true });
Name | Description | Default value |
---|---|---|
audio | If false , then the client will not receive spoken audio from the characters (text only mode). | true |
emotions | You will receive character emotions. | false |
interruptions | Allow interruptions to conversations with the character. | false |
phonemes | Include phoneme information in Audio Events. | false |
silence | Allow for pauses between character replies. | false |
narratedActions | Allow to receive information about character narrated actions | false |
Connection configuration
const connection = {
disconnectTimeout: 10 * 1000, // time in milliseconds
autoReconnect: false, // true by default
}
Name | Description | Default value |
---|---|---|
disconnectTimeout | Close connection due to inactivity | 60000 |
autoReconnect | Connection will be opened automatically on send if closed, or else an existing open connection will be used. Our server closes the connection automatically after 1 minute of inactivity. | true |
SessionToken
It is unsafe to use an API KEY directly on the client side so it is better to have a server generate the token in a safe way. The source code below can be used for this purpose.
async function generateSessionToken() {
const response = await fetch(config.GENERATE_TOKEN_URL);
return response.json();
}
InworldClient
const client = new InworldClient();
// User is not required.
client.setUser(user);
// Configuration is not required.
client.setConfiguration({
capabilities,
connection: {
autoReconnect: false,
audioPlayback: {
// Adjust the volume setting from 1 to 0 when muting, stopping, or experiencing interruptions.
stop: {
// Duration for audio playback cessation, measured in milliseconds.
// The default value is 500 milliseconds.
duration: 500,
// Count of ticks.
// Default value is 25.
ticks: 25,
},
},
// Time in millisecond.
disconnectTimeout: 10 * 1000,
// Specify host for local debugging of the package.
gateway: {
hostname: 'hostname:port',
ssl: true,
},
}
});
// An API key is required to generate a session token.
// It is unsafe to use this key directly on the client side so it is better to have server that can generate the token in a safe way.
// Use Node.js SDK to generate the session token.
// You need to fetch the token using the Node.js service.
client.setGenerateSessionToken(fn);
// It should be like workspaces/{WORKSPACE_NAME}/characters/{CHARACTER_NAME}.
// Or like workspaces/{WORKSPACE_NAME}/scenes/{SCENE_NAME}.
client.setScene(scene);
// Main event handlers
client.setOnDisconnect(fn);
client.setOnError(fn);
client.setOnMessage(fn);
client.setOnReady(fn);
// History change handler.
// History contains all incoming and outgoing events.
// Just display the items on the web page one by one.
client.setOnHistoryChange((history: HistoryItem[]) => {
console.log(history);
});
// Audio event handlers.
client.setOnBeforePlaying((packet: InworldPacket) => {
// Do something with the packet before playing.
});
client.setOnAfterPlaying((packet: InworldPacket) => {
// Do something with the packet after playing.
});
client.setOnStopPlaying(() => {
// Do something when all interaction packages are played.
});
// Finish connection configuration.
// Return instance of EventService.
// Сonnection is not open here. It is just ready to open on message send.
const connection = client.build();
InworldConnectionService
const connection = client.build();
// Open the connection manually. This is available only if configuration.connection.autoReconnect = false.
// Otherwise the connection will be managed automatically by the SDK.
connection.open();
// You can check if the connection is open or not in the case of configuration.connection.autoReconnect = false.
connection.isActive();
// Send a message.
connection.sendText(message);
// Send trigger. Pass trigger name and parameters as arguments.
interface TriggerParameter {
name: string;
value: string;
}
connection.sendTrigger(name: string, parameters?: TriggerParameter[]);
// Send narrated action.
connection.sendNarratedAction(text: string);
// Send an audio start event before call sendAudio.
connection.sendAudioSessionStart();
// Send an audio end event after all audio chunks that you would like to send.
connection.sendAudioSessionEnd();
// Send an audio. Pass string chunk as argument.
connection.sendAudio(chunk);
// Send a cancel response.
// InteractionId or utteranceId can be omitted.
// When interactionId is empty, everything in the session will be removed.
// When only the utteranceId is provided, nothing happens.
// When only the interactionId is provided, everything until this interaction will be removed.
// When both the interactionId and utteranceId are provided, everything until this interaction will be removed, and utterances in this interaction will also be removed.
connection.sendCancelResponse({
interactionId?: string,
utteranceId?: string[],
});
// Close the connection.
connection.close();
// Get a character list.
connection.getCharacters();
// Get the current character.
connection.getCurrentCharacter();
// Change the character in the scene.
connection.setCurrentCharacter(character);
User
const user = {
id: 'user-id',
fullName: 'FirstName LastName',
profile: {
fields: [{ id: 'field_1', value: 'value_1' }]
},
};
// Globally unique string, id of the end user of the system.
// UUID will be used by default.
user.id
// User name.
user.fullName
// List of user profile fields.
user.profile.fields
Character
const character = connection.getCurrentCharacter();
// Character id.
character.id
// Character resource name.
character.resourceName
// Character display name.
character.displayName
// Character assets.
character.assets.avatarImg
character.assets.avatarImgOriginal
character.assets.rpmModelUri
character.assets.rpmImageUriPortrait
character.assets.rpmImageUriPosture
InworldPacket
client.setOnMessage((packet: InworldPacket) => {...});
// It is a text event.
packet.isText();
// It is an audio event.
packet.isAudio();
// It is a trigger event.
packet.isTrigger();
// It is an emotion event.
packet.isEmotion();
// It is a silence event.
packet.isSilence();
// It's a narraged action event.
packet.isNarratedAction();
// It is a control event.
packet.isControl();
// It is a special control event, indicating that interaction has ended.
packet.isInteractionEnd();
// It is a mute control event.
packet.isTTSPlaybackMute();
// It is a unmute control event.
packet.isTTSPlaybackUnmute();
Common Event Data
// ISO string.
packet.date
// A token that uniquely identifies the packet.
packet.packetId.packetId
// A token that uniquely identifies and groups utterances in the replies.
// Different packets may belong to the same utterance. E.g. Audio Event and Text Event of the same spoken utterance.
packet.packetId.utteranceId
// A token that uniquely identifies interaction between actors.
packet.packetId.interactionId
// Determines who sends the packet: player or character.
packet.routing.source.name
packet.routing.source.isPlayer
packet.routing.source.isCharacter
// Determines who receives the packet: player or character.
packet.routing.target.name
packet.routing.target.isPlayer
packet.routing.target.isCharacter
Text Event
// Get message of text event.
packet.text.text
// If this is the final version of the text or not.
// For instance speech recognition may take some time to finalize the text.
packet.text.final
Audio Event
// Get chunk of audio event.
packet.audio.chunk
// Get chunk duration in milliseconds.
// Provided by Web API.
// Available only when metadata is loaded in onBeforePlaying and onAfterPlaying callbacks.
// See setOnBeforePlaying and setOnAfterPlaying handlers.
packet.audio.durationMs
// Get list if phonemes.
packet.audio.additionalPhonemeInfo = [];
// Get phoneme data.
// Synthesized phoneme.
phonemeInfo.phoneme
// Offset from the beginning of audio segment (in seconds).
phonemeInfo.startOffsetS
Emotion Event
// Get behavior affected by emotions.
packet.emotion.behavior.code.NEUTRAL
packet.emotion.behavior.code.DISGUST
packet.emotion.behavior.code.CONTEMPT
packet.emotion.behavior.code.BELLIGERENCE
packet.emotion.behavior.code.DOMINEERING
packet.emotion.behavior.code.CRITICISM
packet.emotion.behavior.code.ANGER
packet.emotion.behavior.code.TENSION
packet.emotion.behavior.code.TENSE_HUMOR
packet.emotion.behavior.code.DEFENSIVENESS
packet.emotion.behavior.code.WHINING
packet.emotion.behavior.code.SADNESS
packet.emotion.behavior.code.STONEWALLING
packet.emotion.behavior.code.INTEREST
packet.emotion.behavior.code.VALIDATION
packet.emotion.behavior.code.AFFECTION
packet.emotion.behavior.code.HUMOR
packet.emotion.behavior.code.SURPRISE
packet.emotion.behavior.code.JOY
// Get strength of the emotion.
packet.emotion.strength.code.UNSPECIFIED
packet.emotion.strength.code.WEAK
packet.emotion.strength.code.STRONG
packet.emotion.strength.code.NORMAL
Trigger Event
// Trigger name.
packet.trigger.name
// Parameters that come with given event.
packet.parameters
// Parameter name.
parameter.name
// Parameter value.
parameter.value
Control Event
packet.control.type.INTERACTION_END
packet.control.type.TTS_PLAYBACK_MUTE
packet.control.type.TTS_PLAYBACK_UNMUTE
Silence Event
// Silence duration in milliseconds.
packet.silence.durationMs
Narrated Action Event
packet.narratedAction.text
Cancel Response Event
// Interaction id for response cancellation.
packet.cancelResponses.interaction_id
// Utterance ids to cancel for the given interaction id.
packet.cancelResponses.utterance_id
Interaction End
Interaction End is a type of Control Event to determine when the Character has ended responding to a sent message. It is considered ended when the chat has not started ( there is no actor action yet ) or the last received message is INTERACTION_END.
onMessage: (inworldPacket: InworldPacket) => {
if (inworldPacket.isInteractionEnd()) {
// Handle end of response
}
}
Determining which interaction has ended for multiple sent messages can be done by comparing the interactionId
, returned when a Player sends a message, to the interactionId
received from a responding message event.
const sendPacketA = await connection.sendText('Hi');
const sendPacketB = await connection.sendText('How are you?');
onMessage: (inworldPacket: InworldPacket) => {
if (inworldPacket.isInteractionEnd()) {
if (sendPacketA.packetId.interactionId == inworldPacket.packetId.interactionId) {
// Handle end of response for sent messsage 'Hi'
}
if (sendPacketB.packetId.interactionId == inworldPacket.packetId.interactionId) {
// Handle end of response for sent messsage 'How are you?'
}
}
}
Session management
Session is expired after 30 minutes of inactivity, resulting in the loss of previous conversation history. Nonetheless, there's a solution to preserve this history.
Previos dialog
You can store all text messages in a location of your preference, allowing you to transfer saved messages to a new session.
For instance, consider the following conversation:
Player: 'Hi'
Character: 'Hi, Player.'
Character: 'How can I assist you today?'
You can save it in an array like this:
const previousDialog = [
{
talker: DialogParticipant.PLAYER,
phrase: 'Hi',
},
{
talker: DialogParticipant.CHARACTER,
phrase: 'Hi, Player. How can I assist you today?',
},
];
To maintain this conversation after the session has expired, you should establish a new connection. Simply use the setSessionContinuation
method and provide the previousDialog
to new connection as shown below:
import { InworldClient, DialogParticipant } from '@inworld/web-core';
const client = new InworldClient();
client.setSessionContinuation({ previousDialog });
Previous state
You can retrieve the conversation state from the Inworld AI server using the following method:
import { InworldClient } from '@inworld/web-core';
let previousState: string;
const client = new InworldClient();
client.setOnDisconnect(async () => {
previousState = (await connection.getSessionState()).state;
});
If you'd like to carry on the conversation with the previous state after the session has ended, you'll need to establish a new connection. Simply use the setSessionContinuation
method for new connection and propagate the previousState
as demonstrated below:
import { InworldClient } from '@inworld/web-core';
const client = new InworldClient();
client.setSessionContinuation({ previousState });
Known Issues
- To avoid any potential issues with browser audio autoplay support you should send audio packets only after the user has initiated some kind of action on UI (e.g., pressing a button).