CLI examples
- Automatic reconnect - Open connection automatically on message send.
- Manual reconnect - Open connection manually before message send.
- Interruptions - Allow for interruptions during the conversation.
Find source code of examples here.
Requirements
- Ensure that the Sox utility is installed and accessible in your PATH. For Windows 10/11, please install version 14.4.1. For other platforms, you can use the latest available version.
How to combine events with different types
In most cases, a user's input can generate various types of events, including text, audio, emotions, and more. To apply multiple types of events simultaneously, such as playing audio and displaying text, you can connect them using the package.packageId.utteranceId
.
Automatic reconnect
Connection will be opened automatically on send if closed, or else an existing open connection will be used.
CLI interface
You will need to use the following console commands to start the client with auto reconnect:
|- /start - starts audio capturing.
|- /end - ends audio capturing.
|- /trigger - send trigger.
|- /info - shows current character.
|- /list-all - shows available characters (created within the scene).
|- /character %character-id% - id of the target character (Get full list using /list-all command).
|- c - cancel current response.
|- <any_other_text> - sends text event to server.
Setup connection
const client = new InworldClient()
.setApiKey({
key: process.env.INWORLD_KEY!,
secret: process.env.INWORLD_SECRET!,
})
.setScene(process.env.INWORLD_SCENE!)
.setOnError((err: ServiceError) => console.error(`Error: ${err.message}`))
.setOnDisconnect(() => console.log('Disconnected'))
.setOnMessage(() => console.log('Message was received'))
.build();
}
Get list of characters and select one of them
If the scene contains more than one character, then the first one will be used by default. You can change this manually by running:
const characters = await connection.getCharacters();
const character = characters.find((c) => c.id === id);
connection.setCurrentCharacter(character);
This should produce the following console output,
/info
Character: 4bafd052-fe3e-483b-9eaf-ab20cadf84b2 (workspaces/test-workspace/characters/character1:First character)
------------------------------
/list-all
0: 4bafd052-fe3e-483b-9eaf-ab20cadf84b2 (workspaces/test-workspace/characters/character1:First character)
1: 82825e9c-7d80-42a4-b6b2-f313c1ac29ed (workspaces/test-workspace/characters/character2:Second character)
------------------------------
/character 82825e9c-7d80-42a4-b6b2-f313c1ac29ed
Character: 82825e9c-7d80-42a4-b6b2-f313c1ac29ed (workspaces/test-workspace/characters/character2:Second character)
------------------------------
Sending a text message, trigger, or cancelling the response to a character
The connection will be opened automatically on send if it is closed, or an existing open connection will be used. Our server will close the connection automatically in one minute if there is inactivity.
await connection.sendText('Hello');
await connection.sendTrigger('Some action');
await connection.sendCancelResponse();
Sending audio chunk to character
Initialize recorder and send audio on each data chunk that it receives. Before beginning capture, send an audio session start
event to the connection. Do not forget to send an audio session end
event at the end of the capture.
await connection.sendAudioSessionStart();
// Start audio capturing here
await connection.sendAudio(chunk);
...
// Stop audio capturing here
await connection.sendAudioSessionEnd();
This should produce the following console output:
/start
Recognized: How are you?, final=true
Character(aa1c8634-df62-4d8b-94eb-4188cf8fbea7) to Player (i=599d9d2e-9994-4208-bb52-b8d5d8997819, u=620d9711-42d4-433f-b5e9-455ac0b7f6aa): I\'m good, thanks!
Character(aa1c8634-df62-4d8b-94eb-4188cf8fbea7) to Player (i=211644a2-2657-40be-b6e6-b88194604717, u=606c9cf7-334b-43e6-af4e-269b78083a87): Thank you for asking.
/end
Close connection
You can close the connection manually if it is not currently required. Be sure to do this on your application stop, e.g., ‘SIGINT’ event. It will be opened automatically when the next packet is sent.
connection.close();
Manual reconnect
If the connection is closed, you will need to open it manually before sending any data.
CLI interface
You will need to use the following console commands to start the client with auto reconnect:
|- /open - open connection.
|- /close - close connection.
|- /start - starts audio capturing.
|- /end - ends audio capturing.
|- /trigger - send trigger.
|- /info - shows current character.
|- /list-all - shows available characters (created within the scene).
|- /character %character-id% - id of the target character (Get full list using /list-all command).
|- c - cancel current response.
|- <any_other_text> - sends text event to server.
Setup connection
If you would like to open and close the connection manually, then you should ensure the property autoReconnect = false
is set in the configuration. It's also possible to specify a disconnectTimeout (1 minute is used by default).
const client = new InworldClient()
.setApiKey({
key: process.env.INWORLD_KEY!,
secret: process.env.INWORLD_SECRET!,
})
.setScene(process.env.INWORLD_SCENE!)
.setConfiguration({
connection: {
autoReconnect: false,
disconnectTimeout: 30 * 1000, // 30 seconds
},
})
.setOnError((err: ServiceError) => console.error(`Error: ${err.message}`))
.setOnDisconnect(() => console.log('Disconnected'))
.setOnMessage(() => console.log('Message was received'))
.build();
}
Open connection
You should open the connection and manage it manually as follows,
if (!connection.isActive()) {
await connection.open();
}
// Do something here
Send message to character
If the connection is open, a message will be sent to the character. Otherwise, you will receive an error message.
await connection.sendText('Hello');
Close connection
You can close the connection manually if it is not currently required. Be sure to do this on your application stop, e.g., ‘SIGINT’ event. It can be opened later.
connection.close();
Interruptions
It's possible for the character to be interrupted during a conversation. To enable this feature, you need to add the interruptions
capability to the character configuration."
CLI interface
You will need to use the following console commands to start the client with interruptions:
|- /start - starts audio capturing.
|- /end - ends audio capturing.
|- /info - shows current character.
|- /list-all - shows available characters (created within the scene).
|- /character %character-id% - id of the target character (Get full list using /list-all command).
Setup connection
Don't forget to set up the interruptions
capability.
const client = new InworldClient()
.setApiKey({
key: process.env.INWORLD_KEY!,
secret: process.env.INWORLD_SECRET!,
})
.setConfiguration({
capabilities: { interruptions: true },
})
.setScene(process.env.INWORLD_SCENE!)
.setOnError((err: ServiceError) => console.error(`Error: ${err.message}`))
.setOnDisconnect(() => console.log('Disconnected'))
.setOnMessage(() => console.log('Message was received'))
.build();
}
Messages management
During a character's speech (i.e., when audio packets are playing), the user can interrupt the character. This occurs in the following situation: when the user has said something and the text has been recognized (you can use packet.routing.source.isPlayer
to detect it).
- When the recognition event is received, ensure if the character is currently speaking by checking the audio packet queue for the same
interactionId
. - If the character is speaking, remove all audio and text events with the same
interactionId
from the queue. - Call the
sendCancelResponse
method with theutteranceId
andinteractionId
to cancel any events that were not applied on the server-side. - If the character is not currently speaking, do nothing and continue playing the audio packets.
Please refer to the provided source code for implementation details.
Handlers
onDisconnect
Each time the server closes the connection or you call connection.close
, a disconnect event will be triggered.
connection.setOnDisconnect(() => console.log('Disconnected'));
onError
connection.setOnError((err: ServiceError) => console.error(`Error: ${err.message}`));
onMessage
To detect the type of message and print or play, you can use,
connection.setOnMessage(async (packet: InworldPacket) => {
const { packetId } = packet;
const i = packetId.packetId;
const u = packetId.utteranceId;
// TEXT
if (packet.isText()) {
const textEvent = packet.text;
if (packet.routing.source.isPlayer) {
if (textEvent.final) {
console.log(
`Recognized: ${textEvent.text}, final=${textEvent.final}`,
);
}
} else {
console.log(
`${this.renderEventRouting(packet)} (i=${i}, u=${u}): ${
textEvent.text
}`,
);
}
}
// AUDIO
if (packet.isAudio()) {
player.send(packet.audio.chunk);
}
});