Graph Variants & Experimentation

​

Understanding Variants

• Deploy independently while sharing the same endpoint
• A/B test against other variants
• Roll back to if needed
• Track performance metrics for

​

Variant Use Cases

• Test GPT-4 vs Claude vs Gemini
• Compare different model parameters (temperature, token limits)
• Evaluate custom vs standard model configurations

• A/B test different system prompts
• Test various conversation styles
• Optimize for specific use cases or user segments

• Test different node configurations
• Compare processing pipelines
• Optimize for performance vs quality trade-offs

​

Working with Variants

​

Prerequisites

// CRITICAL: Enable remote config in your graph code
const graph = new GraphBuilder({
  id: 'my-graph-id',  // Must match CLI deployment
  apiKey: process.env.INWORLD_API_KEY,
  enableRemoteConfig: true, // Required for Portal experiments
})
  .addNode(llmNode)
  .setStartNode(llmNode)
  .setEndNode(llmNode)
  .build();

​

Variant Workflow

1. Preview your graph structure

inworld graph variant print ./graph.ts

2. Deploy your graph (if not already deployed)

inworld deploy ./graph.ts  

3. Register a variant in Portal

inworld graph variant register -d [variant-name] ./graph.ts

​

Variant Examples & Use Cases

​

Create a Baseline Variant

inworld graph variant register -d baseline ./graph.ts

​

Register an Experimental Variant

inworld graph variant register -d experiment-v2 ./graph-v2.ts

​

Version-based Variants

inworld graph variant register -d v1.2.0 ./graph.ts

​

Update Existing Variant

# Force update if variant already exists
inworld graph variant register -d baseline ./graph.ts --force

​

Variant Management

​

List All Variants

inworld graph variant list ./graph.ts

​

Graph Registry Integration

​

Portal Setup

1. Navigate to Graph Registry in Portal
2. Click Register Graph
3. Enter your graph ID (same as in CLI: my-graph-id)
4. Click Register

• Create targeting rules based on user attributes
• Set traffic distribution percentages
• Enable/disable experiments
• Monitor performance metrics

​

CLI Variant Registration

# Register variants via CLI
inworld graph variant register -d baseline ./graph.ts
inworld graph variant register -d experiment-v1 ./improved-graph.ts

​

User Context for Proper Experimentation

# CORRECT: Include UserContext for proper A/B testing
inworld run ./graph.ts '{
  "input": {"message": "Hello"},
  "userContext": {
    "targetingKey": "user123",
    "attributes": {
      "country": "US", 
      "tier": "premium",
      "app_version": "2.1.0"
    }
  }
}'

​

UserContext Best Practices

// In your application code
const userContext = new UserContext({
  tier: user.tier,
  country: user.country,
  app_version: "2.1.0"
}, user.id); // Unique targeting key

graph.start(input, userContext);

• User Demographics: age, country, language
• Subscription Info: tier (free, premium), plan_type
• Technical: app_version, platform (web, mobile)
• Behavioral: usage_frequency, feature_usage

​

Experimentation Workflow

​

Complete A/B Testing Process

1. Deploy initial graph → Creates persistent endpoint using CLI
2. Register as baseline variant → Use CLI to register initial version
3. Create experimental changes → Modify your graph locally
4. Register experimental variant → Use CLI to register new version
5. Manage traffic in Portal → Configure A/B testing splits in web interface
6. Monitor results in Portal → View performance metrics and user feedback
7. Iterate with CLI → Register new variants as you develop
8. Scale via Portal → Adjust traffic distribution to winning variants

​

Experiment Design Best Practices

• Calculate sample size upfront - Use Power Analysis calculator with baseline metrics, desired MDE (typically 2-5%), α = 0.05, and power = 0.80
• Create variants with consistent naming - Use naming conventions like model-prompt-tools (e.g., “GPT5-Creative-Memory” vs “Claude4Sonnet-Analytical-RAG”)
• Always pass UserContext with targeting keys - Use unique user IDs as targeting keys to ensure consistent variant assignment

• Start with small traffic allocation - Use 10-20% traffic to validate setup, then scale to reach calculated sample size
• Use rule ordering strategically - Put specific targeting rules (premium users, specific regions) at the top since rules evaluate top-to-bottom
• Monitor via Portal dashboards - Watch custom metrics alongside default metrics (Graph Executions Total, P99 Latency)

• Wait for statistical significance before making decisions
• Use gradual rollout - Deploy winners by increasing traffic allocation (50/50 → 70/30 → 90/10)
• Clean up experiments properly - Disable and delete old rules to keep Graph Registry organized

​

Production Variant Management

​

Scaling Winning Variants

1. Gradual Rollout: Increase traffic to winner gradually (30% → 50% → 70% → 90%)
2. Monitor Metrics: Watch for any degradation during rollout
3. Full Migration: Set winning variant as default for new users
4. Cleanup: Retire losing variants and clean up Portal rules

​

Variant Lifecycle

• Create feature branches for experimental variants
• Test locally before registering variants
• Use descriptive naming conventions

• Register variants with CLI
• Configure experiments in Portal
• Monitor performance and user feedback

• Scale winning variants gradually
• Archive losing variants for future reference
• Document learnings and results

​

Long-term Management

• Regularly review and clean up old variants
• Archive successful experiments for reference
• Maintain naming conventions and documentation
• Set up alerts for experiment performance

• Plan regular experimentation cycles
• Test incremental improvements continuously
• Use learnings to inform future variant development
• Build experimentation into your development process

​

Next Steps

1. Advanced Integration - Complete Portal integration and monitoring setup
2. Portal Graph Registry - Learn advanced experiment configuration
3. A/B Experiment Best Practices - Deep dive into experimentation methodology

​

Having Issues?