Overview - Inworld AI Documentation

In the Unity AI Runtime, you can use the 3 following ways to interact with Inworld AI modals:

Using the C# code to directly access the Primitive Modules.
Generating a ScriptableObject GraphAsset, with the also generated EdgeAssets and NodeAssets that will interact with primitives automatically.
Using the GraphViewEditor to automatically generate the graph system.

Apparently, using the GraphViewEditor should be the easiet. Although in some case you will still need to directly modify the ScriptableObjects or even the Code. For those part, we will introduce later. In this tutorial, let’s focus on creating graphs with GraphViewEditor.

Life Cycle

For the graph/node system, the life cycle of the runtime (When you actually click the Play button) is like this:

You will need a InworldGraphExecutor to hold the GraphAsset, it will do the above steps. You will also need a NodeTemplate to communicate with the InworldGraphExecutor, to send and receive the data from Inworld.

Clone a Scene

In this overview, let’s focus on the graph creating, so we will use the current default GraphNodeTemplate in the demos. Let’s clone a scene from Assets/InworldRuntime/Scenes/Nodes/CharacterInteractionNode.unity.Let’s rename the cloned one whatever you want. i.e; Demo, and open it.

Check out the current Graph

The graph is held under the InworldGraphExecutor, which is at the root hierarchy of CharacterInteractionCanvas.Navigate to that place, double click the asset to open it in Inspector.

Scroll down until you see the Open Graph Editor button. Click it.

You’ll see the graph in the editor, and you’ll have a glimpse about how each node works together.

You can scroll the mouse wheel to zoom in and zoom out.You can press and hold the mouse wheel to move around.

Let’s not edit this graph to avoid polluting the demo. We will create our own later.

Create your own Graph Asset

Create Graph

Right click on the Project tab, select Create > Inworld > Create Graph > CharacterInteraction.

Rename and save the asset.

In the inspector, fill in the essential data. User data and character data are essential.

User Data
Character Data (In Characters list of the Character Interaction Data).
Voice (If you’re planning to use TTS)
Prompt (if you’re planning to use LLM)

You can click the ◎ icon to select the default one first.Then clone and update your own data later.

Edit your Graph

Open Graph Editor

Once the character data and the user data has been set, let’s click the Open in Graph Editor button.

Create a FilterInputNode

Right click on the Graph Editor, select Create Node > Custom Nodes... > FilterInputNode.Rename it to Input.

Each node has to be renamed and has a unique name after created. The graph compiler will fail if it found duplicated names.

The FilterInputNode is good to be use as the StartNode, because you can use it to receive multiple input nodes.For example, currently it will detect if the format of the inputting data is correct.If the Inputting data is InworldText or InworldAudio, it will pass, otherwise it will fail.

Currently, this node only support InworldText and InworldAudio. But please feel free to inherit this FilterInputNodeAsset and add your desired input type on your own. (Still must be a existing child class of InworldBaseData though)

Create a STTNode

From the output of FilterInput, hold left mouse button, drag an edge out.Create a STTNode and rename it to STT.STT(Speech-to-text) Node is used to generate the text from player’s audio input.It accepts InworldAudio as input, sends InworldText as output.

Did you notice that the background color of the STTNode is different?

That’s because it’s a dll node, not a custom node.You probably don’t want to inherit a dll node, because if you do, the graph system may get broken.

Create a AddSpeechEventNode

Right click on the Graph Editor, select Create Node > Custom Nodes... > AddSpeechEventNode.Rename it to be PlayerSpeech.Set the IsPlayer to be true.The AddSpeechEventNode basically converts everything to InworldText.Meanwhile, it will store this piece of data in the ConversationData of the PromptAsset, which will later be used to generate the prompt.

It’s expected to have a pair of AddSpeechEventNode, one for player, one for agent.By setting the IsPlayer to be true, the generated data will be set into the prompt as the Player's speech.If not set, the agent will get confused with its identity.

Connect Nodes

Connect Input to STT, STT to PlayerSpeech, and Input to PlayerSpeech.\

Change Edge Types

Right click the edge from Input to STT, select Set Edge Type > Audio.Right click the edge from Input to PlayerSpeech, select Set Edge Type > Text.

By default, the edge can pass through everything without checking.
But each node can only receive certain amount of data types as their input.
Data type mismatch may even lead the Unity Editor to crash!
So it’s necessary to set data type for the edges to mark specific data transfer.

Connect to FormatPromptNode

Extend PlayerSpeech node out to create a FormatPromptNodeThe FormatPromptNode is a customNode. It converts InworldText to LLMChatRequest, that will be used in LLMNode as input.The FormatPromptNode will use the current graph’s prompt data. Set under the InworldGraphAsset you just filled.

If you double click to open it, you’ll find 2 fields. Prompt and JinjaPrompt.We will use jinja to take the previous node’s output, InworldText, together with the previously filled UserData, CharacterData to replace those {{}} defined the Prompt to generate the JinjaPrompt as the result.

And the JinjaPrompt, is the actual data we are going to send to LLMNode

Connect to LLMNode

Extend FormatPromptNode node out to create a LLMNode.The LLMNode is a dll node that cannot be inherrited.It takes LLMChatRequest as input, returns LLMChatResponse as output.

Create another AddSpeechEventNode for Character

Extend LLMNode node out to create another AddSpeechEventNode.Rename it to be CharacterSpeech.Set the IsPlayer to be false.

Connect to TTSNode

Extend CharacterSpeech node out to create a TTSNode.Rename it to be TTS.Set the voice to be an available voiceID (Default is not an available ID right now. Please rename a valid voiceID)

Done!You’ve finally finished a basic character conversation graph, with InworldText and InworldAudio as input, InworldAudio as output.It will take your typed or spoken speech as input, use STT to convert your audio into text, pasted in the current prompt, get the response, then speak out with TTS.

Save your graph

You can simply press Ctrl + S or press the Save button at the top to save this graph.

After saved, you will find all the related NodeAssets and EdgeAssets are already created.And all the Start/End Nodes are set.Those ScriptableObjects are stored by default under the folder Asset/Data/{GraphName}

Test your Graph

Apply your graph

Find the InworldGraphExecutor, by default it’s at the CharacterInteractionCanvas.Replace the graph asset with the one you just created.

Run in editor

You will find generally it’s working, just there are not bubbles popping out.That’s because we only have 1 end node, that’s a TTSNode, so only the TTSOutput will be send out during the graph execution.

Update your Graph

Add ConversationEndpointNodes

Open your graph editor, adding ConversationEndpointNode right after each AddSpeechEventNode (CharacterSpeech and PlayerSpeech)The ConversationEndpointNodes is a customNode that often used as EndNodes, it will generate the sentence like {Speaker}: {Contents}.For the user to extract and render correspondently.

Rename ConversationEndpointNodes

Rename the one after CharacterSpeech to CharacterFinal and set the IsPlayer to be false.
Rename the one after PlayerSpeech to PlayerFinal and set the IsPlayer to be true.

Save the graph

Press “Save”, you will see the data of the GraphAsset has changed.The EndNodes now contains 3 nodes.\

Run in editor

Done! You will find generally it’s working, and the bubbles are also popped out!

​Life Cycle

Clone a Scene

Check out the current Graph

Create your own Graph Asset

Edit your Graph

Test your Graph

Update your Graph

Life Cycle