Experiments

​

Summary

1. Code your agent – Code your agent, enable remote config, and deploy via CLI.
2. Register variants – Use inworld graph variant register to register the graph for experiment. Or upload the JSON via UI.
3. Start experiment – Set targeting + rollout percentages in the Experiments Tab in Portal
4. Monitor & roll out – Monitor metrics dashboards, then promote the winner.

​

Experiment workflow

​

Step 1 – Code your agent

npm install -g @inworld/cli

inworld init --template llm-to-tts-node --name my-agent
cd my-agent

const graph = new GraphBuilder({
  id: 'my-graph-id',
  apiKey: process.env.INWORLD_API_KEY,
  enableRemoteConfig: true, // Required for Experiments
})
  .addNode(llmNode)
  .setStartNode(llmNode)
  .setEndNode(llmNode)
  .build();

const userContext = new UserContext({ // Add user context 
  tier: user.tier,
  country: user.country,
  app_version: '2.1.0',
}, user.id); // targeting key

inworld run ./graph.ts '{"input":{"message":"Hello"},"userContext":{"targetingKey":"user123","attributes":{"tier":"premium","country":"US"}}}'

import { telemetry, MetricType } from '@inworld/runtime';

telemetry.init({
  apiKey: process.env.INWORLD_API_KEY,
  appName: 'cli-ops',
  appVersion: '1.0.0',
});

telemetry.configureMetric({
  metricType: MetricType.COUNTER_UINT,
  name: 'successful_interactions',
  description: 'Count of responses that reached business success criteria',
});

​

Step 2 – Register variants

• Hosted endpoint (recommended): Deploying with the CLI automatically registers the graph ID and baseline variant:

inworld deploy ./graph.ts

• Self-host + CLI: After building your graph, push the baseline configuration manually:

inworld graph variant register -d baseline ./graph.ts

• Self-host + UI: Export the graph JSON and upload it in Portal:

inworld graph variant print ./graph.ts > baseline.json

1. Open Register Graph and enter the graph ID exactly as defined in your code.\
2. Click Create Variant, name it, and upload baseline.json.\

cp graph.ts graph-claude.ts

const llmNode = new RemoteLLMChatNode({
  provider: 'openai',
  modelName: 'gpt-5-mini',
  stream: true,
  textGenerationConfig: { maxNewTokens: 200 },
});

inworld graph variant register -d claude ./graph-claude.ts

inworld graph variant list ./graph.ts

​

Step 3 – Start an experiment

• Add filters on user attributes (user_tier, location, etc.) and assign percentages that sum to 100%.
• Save the rule and repeat for additional cohorts (keep specific rules at the top; use “Everyone else” as the fallback).

• Use the rule menu → Enable, then click Save to go live.
• Start with small allocations (10–20%) to validate; increase once metrics look good.
• Use the same rule menu to disable, delete, duplicate, or drag-to-reorder rules.

​

Step 4 – Monitor & roll out

• Watch metrics, dashboards, traces, and logs while the experiment runs.
• Increase the winning variant’s allocation gradually (50/50 → 70/30 → 90/10), then set it to 100% and retire old rules.
• Roll back or tweak allocations if latency, errors, or business KPIs regress.

​

How Experiments picks variants

1. Remote config must be enabled.
2. The graph ID must be registered in Experiments and have at least one active rule that returns a variant.

1. Local cache check: If the compiled variant for this user is cached, it executes immediately; otherwise Experiments is queried.
2. Variant fetch: Experiments evaluates your targeting rules, returns the selected variant, and falls back to the local configuration if no rule applies or the fetch fails.
3. Compile & cache: The runtime compiles the variant payload, caches it, and executes the graph with the new configuration.

​

Troubleshooting

1. Specify a targeting key: This is typically the user ID and ensures the same user gets consistent variants.
   Copy

   // CORRECT: Each user gets a consistent variant
   const userContext = new UserContext(
       {  
           country: country  // attributes for targeting rules
       }, 
       userId  // targeting key
   );
   await graphExecutor.execute(input, executionId, userContext);
   

   Copy

   // INCORRECT: All users get the same variant
   await graphExecutor.execute(input, executionId); 
   

2. Include targeting attributes: Make sure to pass any attributes that your targeting rules use (e.g., country, user_tier, etc.).

• Ensure remote config is enabled and the graph ID is registered in Experiments.
• Verify that the relevant targeting rules are enabled and percentages sum to 100%.
• Pass different user attributes to confirm that variant assignments change as expected.
• Monitor logs/dashboards for variant metadata or assignment info.

1. Missing INWORLD_API_KEY so the graph never authenticates.
2. enableRemoteConfig is set to false.
3. Graph not registered in Experiments.
4. No active targeting rules or no variants configured for the matched cohort.

• Switch LLM/STT/TTS models or providers.
• Adjust node configuration (temperature, token limits, prompts).
• Reorder/add/remove built-in nodes while preserving the same inputs/outputs.
• Update processing logic (edge conditions, preprocessing steps, data flow).

1. Adding new component types.
2. Introducing unregistered custom nodes.
3. Changing the graph’s input/output interface.
4. Using custom edge conditions beyond supported CEL expressions.