> ## Documentation Index
> Fetch the complete documentation index at: https://docs.inworld.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Overview
In the Unity AI Runtime, you can use the 3 following ways to interact with Inworld AI modals:
* Using the C# code to directly access the Primitive Modules.
* Generating a ScriptableObject GraphAsset, with the also generated EdgeAssets and NodeAssets that will interact with primitives automatically.
* Using the GraphViewEditor to automatically generate the graph system.
Apparently, using the GraphViewEditor should be the easiet.
Although in some case you will still need to directly modify the ScriptableObjects or even the Code.
For those part, we will introduce later.
In this tutorial, let's focus on creating graphs with GraphViewEditor.
## Life Cycle
For the graph/node system, the life cycle of the runtime (When you actually click the `Play` button) is like this:
You will need a `InworldGraphExecutor` to hold the `GraphAsset`, it will do the above steps.
You will also need a `NodeTemplate` to communicate with the `InworldGraphExecutor`, to send and receive the data from Inworld.
In this overview, let's focus on the graph creating, so we will use the current default GraphNodeTemplate in the demos.
Let's clone a scene from `Assets/InworldRuntime/Scenes/Nodes/CharacterInteractionNode.unity`.
Let's rename the cloned one whatever you want. i.e; `Demo`, and open it.
The graph is held under the `InworldGraphExecutor`, which is at the root hierarchy of `CharacterInteractionCanvas`.
Navigate to that place, double click the asset to open it in Inspector.
Scroll down until you see the `Open Graph Editor` button. Click it.
You'll see the graph in the editor, and you'll have a glimpse about how each node works together.
You can scroll the mouse wheel to zoom in and zoom out.
You can press and hold the mouse wheel to move around.
Let's not edit this graph to avoid polluting the demo. We will create our own later.
Right click on the `Project` tab, select `Create > Inworld > Create Graph > CharacterInteraction`.
* User Data
* Character Data (In Characters list of the Character Interaction Data).
* Voice (If you're planning to use TTS)
* Prompt (if you're planning to use LLM)
You can click the `◎` icon to select the default one first.
Then clone and update your own data later.
Once the character data and the user data has been set, let's click the `Open in Graph Editor` button.
Right click on the Graph Editor, select `Create Node > Custom Nodes... > FilterInputNode`.
Rename it to `Input`.
Each node has to be renamed and has a unique name after created.
The graph compiler will fail if it found duplicated names.
The `FilterInputNode` is good to be use as the StartNode, because you can use it to receive multiple input nodes.
For example, currently it will detect if the format of the inputting data is correct.
If the Inputting data is `InworldText` or `InworldAudio`, it will pass, otherwise it will fail.
Currently, this node only support `InworldText` and `InworldAudio`.
But please feel free to inherit this `FilterInputNodeAsset` and add your desired input type on your own.
(Still must be a existing child class of InworldBaseData though)
From the output of FilterInput, hold left mouse button, drag an edge out.
Create a `STTNode` and rename it to `STT`.
`STT(Speech-to-text) Node` is used to generate the text from player's audio input.
It accepts `InworldAudio` as input, sends `InworldText` as output.
Did you notice that the background color of the STTNode is different?
That's because it's a dll node, not a custom node.
You probably don't want to inherit a dll node, because if you do, the graph system may get broken.
Right click on the Graph Editor, select `Create Node > Custom Nodes... > AddSpeechEventNode`.
Rename it to be `PlayerSpeech`.
Set the `IsPlayer` to be `true`.
The `AddSpeechEventNode` basically converts everything to `InworldText`.
Meanwhile, it will store this piece of data in the `ConversationData` of the `PromptAsset`, which will later be used to generate the prompt.
It's expected to have a pair of AddSpeechEventNode, one for player, one for agent.
By setting the `IsPlayer` to be true, the generated data will be set into the prompt as the `Player's speech`.
If not set, the agent will get confused with its identity.
Connect `Input` to `STT`, `STT` to `PlayerSpeech`, and `Input` to `PlayerSpeech`.\\
Right click the edge from `Input` to `STT`, select `Set Edge Type > Audio`.
Right click the edge from `Input` to `PlayerSpeech`, select `Set Edge Type > Text`.
By default, the edge can pass through everything without checking.
But each node can only receive certain amount of data types as their input.
Data type mismatch may even lead the Unity Editor to **crash**!
So it's necessary to set data type for the edges to mark specific data transfer.
Extend `PlayerSpeech` node out to create a `FormatPromptNode`
The `FormatPromptNode` is a customNode. It converts `InworldText` to `LLMChatRequest`, that will be used in `LLMNode` as input.
The `FormatPromptNode` will use the current graph's prompt data. Set under the `InworldGraphAsset` you just filled.
If you double click to open it, you'll find 2 fields. `Prompt` and `JinjaPrompt`.
We will use jinja to take the previous node's output, `InworldText`, together with the previously filled `UserData`, `CharacterData` to replace those `{{}}` defined the `Prompt` to generate the `JinjaPrompt` as the result.
And the `JinjaPrompt`, is the actual data we are going to send to `LLMNode`
Extend `FormatPromptNode` node out to create a `LLMNode`.
The LLMNode is a dll node that cannot be inherrited.
It takes `LLMChatRequest` as input, returns `LLMChatResponse` as output.
Extend `LLMNode` node out to create another `AddSpeechEventNode`.
Rename it to be `CharacterSpeech`.
Set the `IsPlayer` to be `false`.
Extend `CharacterSpeech` node out to create a `TTSNode`.
Rename it to be `TTS`.
Set the voice to be an available voiceID (Default is not an available ID right now. Please rename a valid voiceID)
Done!
You've finally finished a basic character conversation graph, with `InworldText` and `InworldAudio` as input, `InworldAudio` as output.
It will take your typed or spoken speech as input, use STT to convert your audio into text, pasted in the current prompt, get the response, then speak out with TTS.
You can simply press `Ctrl + S` or press the `Save button` at the top to save this graph.
After saved, you will find all the related NodeAssets and EdgeAssets are already created.
And all the Start/End Nodes are set.
Those ScriptableObjects are stored by default under the folder `Asset/Data/{GraphName}`
Find the `InworldGraphExecutor`, by default it's at the `CharacterInteractionCanvas`.
Replace the graph asset with the one you just created.
You will find generally it's working, just there are not bubbles popping out.
That's because we only have 1 end node, that's a TTSNode, so only the TTSOutput will be send out during the graph execution.
Open your graph editor, adding `ConversationEndpointNode` right after each `AddSpeechEventNode` (CharacterSpeech and PlayerSpeech)
The `ConversationEndpointNodes` is a customNode that often used as EndNodes, it will generate the sentence like `{Speaker}: {Contents}`.
For the user to extract and render correspondently.
* Rename the one after `CharacterSpeech` to `CharacterFinal` and set the `IsPlayer` to be `false`.
* Rename the one after `PlayerSpeech` to `PlayerFinal` and set the `IsPlayer` to be `true`.
Press "Save", you will see the data of the `GraphAsset` has changed.
The `EndNodes` now contains 3 nodes.\\
Done!
You will find generally it's working, and the bubbles are also popped out!