Realtime TTS-2 is live. Built for realtime conversation that feels human. Read the Realtime TTS-2 announcement

Cost and latency

Prompt compression

Reduce token usage by compressing long messages before sending them to the LLM

Prompt compression shortens long messages before they reach the LLM. It reduces input tokens and cost without meaningfully changing response quality.

Compression itself is not billed today. Input is billed on the compressed prompt: the usage.prompt_tokens in the response counts the tokens the model received after compression, so the tokens saved are tokens you don't pay for.

Compression works best on long system prompts, detailed instructions, and context-heavy messages. The 250-token minimum applies to each run of consecutive messages that share the same aggressiveness, not to each message or to the whole request: Inworld compresses such a run together when its text adds up to 250 tokens or more, and skips it with a warning when it is shorter. With variant- or request-level compression, every compressible message gets the same setting, so a short message next to a long one is compressed with it. A tool-result message, which those settings skip, ends the run.

Compression is also skipped, without a warning, when the first model the request tries has an input price below $0.05 per million tokens (pricing.promptToken below 0.00000005 in the List models API), where the savings would be negligible.

Compression is available on chat completions (/v1/chat/completions). The Responses API (/v1/responses) returns a 400 error for a compression field. It skips a router variant's compression setting and runs the request uncompressed, with a warning that names the skipped setting.

Compression may produce unexpected results on structured content such as JSON, XML, or code. Avoid compressing messages that contain structured data the model needs to parse exactly.

Router-level compression

Set compression on a router variant to compress every request routed to it. This is the most common setup: configure it once for all requests through that variant.

Router with compression
{
  "name": "routers/my-router",
  "display_name": "Compressed prompts",
  "defaultRoute": {
    "route_id": "default",
    "variants": [{
      "weight": 100,
      "variant": {
        "variant_id": "v1",
        "model_id": "openai/gpt-4o",
        "compression": {
          "aggressiveness": 0.7
        },
        "message_templates": [
          {
            "role": "system",
            "content": "You are a helpful assistant with deep expertise in world history, geopolitics, and international relations. Your primary responsibility is to provide detailed, engaging answers..."
          }
        ]
      }
    }]
  }
}
ParameterTypeRangeDefaultDescription
aggressivenessfloat0.0–1.00.5How aggressively to compress. Higher values save more tokens but may reduce fidelity. 0 disables compression.

Variant-level compression applies to every compressible message in the request: the router's message templates and the messages sent in the request, including user messages. Tool-result messages are skipped. When variant-level compression is set, per-message compression settings are ignored.

Request-level compression

To compress a single request the same way, pass compression at the top level of the request body (or inside extra_body). It takes the same aggressiveness parameter and overrides any variant-level setting. Like variant-level compression, it applies to all compressible messages and replaces per-message settings. To turn off a variant's compression for one request, send "compression": {"aggressiveness": 0}.

json
{
  "model": "openai/gpt-4o",
  "messages": [
    { "role": "system", "content": "You are a support agent for Acme. Follow these policies: ..." },
    { "role": "user", "content": "How do I reset my password?" }
  ],
  "compression": { "aggressiveness": 0.6 }
}

Message-level compression

You can compress specific messages in a chat completion request by adding a compression field to any message. Consecutive messages with the same aggressiveness are batched and compressed together. Non-consecutive compressed messages are processed separately.

bash
curl -X POST https://api.inworld.ai/v1/chat/completions \
  -H "Authorization: Basic $INWORLD_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [
      {
        "role": "system",
        "content": "You are an expert assistant with extensive knowledge of...",
        "compression": { "aggressiveness": 0.5 }
      },
      {
        "role": "user",
        "content": "What is the capital of France?"
      }
    ]
  }'

Only messages with the compression field are compressed. Other messages are sent as-is. Per-message settings apply only when neither the variant nor the request sets compression.

Latency and prompt caching

Compression runs before the request goes to the model. Inworld reuses a message's compressed text for up to an hour, keyed by the message text and the aggressiveness value. A message sent again within that hour, such as the same system prompt on every turn, reuses that text without calling the compression service again. Other messages are compressed in batches, several at a time. Each call to the compression service has a one-second timeout, so a request that needs more batches than run at once can take longer. If a batch fails or times out, its messages are sent uncompressed.

Because the same message compresses to the same text during that hour, a compressed system prompt stays a stable prefix, so the provider's implicit caching can still apply.

Compression currently replaces a message's text parts with a single text part. Any cache_control breakpoint on those parts is dropped, so the message is not explicitly cached. Do not compress messages that carry cache_control. Variant- and request-level compression apply to every compressible message, so do not combine them with explicit caching.

Response

Non-streaming responses include compression stats in metadata.compression. Streamed responses do not carry compression stats or warnings.

json
{
  "metadata": {
    "compression": {
      "original_tokens": 190,
      "compressed_tokens": 102,
      "saved_tokens": 88
    }
  }
}
FieldDescription
original_tokensTotal tokens before compression
compressed_tokensTotal tokens after compression
saved_tokensTokens saved (original - compressed)

If no messages were compressed, the compression field is omitted from the response.

Warnings

When compression is enabled but skipped or fails for specific messages, a non-streaming response includes warnings in metadata.compression_warnings:

json
{
  "metadata": {
    "compression_warnings": [
      "Prompt compression skipped for message #0 (system: 'You are a helpful...') — text too short (min 250 tokens)"
    ]
  }
}

Warnings appear when:

  • A message has compression enabled but its run of messages is too short (below 250 tokens)
  • The compression service fails for a message (the message is sent uncompressed)

No warning is returned when compression is skipped because the model's input price is below the minimum, or because the request or router variant sets aggressiveness to 0. If a request you expected to compress has no metadata.compression and no warning, check the model's pricing.promptToken.