Cost and latency

Prompt caching

Reduce AI model costs with implicit and explicit prompt caching

Use prompt caching to save on inference costs where the provider and model support it. Inworld Inference supports implicit and explicit caching.

Implicit Caching

Implicit caching works automatically when the model provider supports it. You do not need to configure it, but the provider's rules apply.

Provider stickiness: Inworld Inference ensures all requests go to the same model provider for a given conversation. If you hit cache with one provider, subsequent requests stay routed to that provider, so you never lose cached data by being switched to a different provider.

Providers such as OpenAI, DeepSeek, and Google Gemini 2.5 typically offer implicit caching. Minimum token counts and TTL behavior vary by provider. Check the provider's documentation for pricing and model-specific details.

Explicit Caching

Explicit caching is supported for Anthropic and Google providers. It gives you control over what gets cached and for how long.

Unified Protocol

Both providers use the same protocol: a cache_control object in each message content part, including a configurable ttl (time to live).

Cache Control in Messages

Add cache_control to text parts within multipart message content. Reserve it for large bodies of text such as character cards, CSV data, RAG context, or book chapters.

json

{
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "You are a historian studying the fall of the Roman Empire. Below is an extensive reference book:"
        },
        {
          "type": "text",
          "text": "HUGE TEXT BODY HERE",
          "cache_control": {
            "type": "ephemeral",
            "ttl": "1h"
          }
        }
      ]
    },
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What triggered the collapse?"
        }
      ]
    }
  ]
}

TTL Management

Configurable TTL: Set ttl in the cache_control object (e.g., "ttl": "5m", "ttl": "1h").
Update at any time: You can change the TTL on subsequent requests if you originally set it longer than needed.
Automatic prolongation: When 5% or less of the TTL remains and there are still messages hitting the cache, Inworld Inference automatically extends the TTL to keep your cache warm.

User Message Example

json

{
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Based on the book text below:"
        },
        {
          "type": "text",
          "text": "HUGE TEXT BODY HERE",
          "cache_control": {
            "type": "ephemeral",
            "ttl": "30m"
          }
        },
        {
          "type": "text",
          "text": "List all main characters mentioned in the text above."
        }
      ]
    }
  ]
}

Inspecting Cache Usage

To see how much caching saved on each generation, check the prompt_tokens_details object in the usage response.

Request and Response Samples

The following samples were verified against the Inworld Inference API. The first request establishes a cache; the second request hits the cache.

Request 1 — Cache write (establishing cache)

bash

curl -X POST "https://api.inworld.ai/v1/chat/completions" \
  -H "Authorization: Basic <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google-ai-studio/gemini-2.5-flash",
    "messages": [
      {
        "role": "system",
        "content": [
          {
            "type": "text",
            "text": "You are a helpful historian. Below is reference material:"
          },
          {
            "type": "text",
            "text": "<large text body of 1024+ tokens>",
            "cache_control": {"type": "ephemeral", "ttl": "5m"}
          }
        ]
      },
      {"role": "user", "content": "What year did the Western Roman Empire fall? Reply in one short sentence."}
    ],
    "max_tokens": 50
  }'

Response 1 — Cache write

json

{
  "id": "chatcmpl-1772169766796",
  "model": "google-ai-studio/gemini-2.5-flash",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The Western Roman Empire fell in 476 AD.\n"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 1402,
    "completion_tokens": 13,
    "total_tokens": 1415,
    "prompt_tokens_details": {
      "cache_write_tokens": 1388,
      "cached_tokens": 1388
    }
  }
}

Request 2 — Cache hit (same conversation, follow-up question)

Send the same system prompt and cached content, plus the previous exchange and a new user message:

bash

curl -X POST "https://api.inworld.ai/v1/chat/completions" \
  -H "Authorization: Basic <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google-ai-studio/gemini-2.5-flash",
    "messages": [
      {
        "role": "system",
        "content": [
          {"type": "text", "text": "You are a helpful historian. Below is reference material:"},
          {
            "type": "text",
            "text": "<same large text body as request 1>",
            "cache_control": {"type": "ephemeral", "ttl": "5m"}
          }
        ]
      },
      {"role": "user", "content": "What year did the Western Roman Empire fall? Reply in one short sentence."},
      {"role": "assistant", "content": "The Western Roman Empire fell in 476 AD.\n"},
      {"role": "user", "content": "Name two factors that contributed to its fall."}
    ],
    "max_tokens": 80
  }'

Response 2 — Cache hit

json

{
  "id": "chatcmpl-1772169775239",
  "model": "google-ai-studio/gemini-2.5-flash",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "While the provided text doesn't explicitly detail the reasons for the fall of the Western Roman Empire, it does highlight factors key to its rise and endurance... Drawing inferences from that, two factors could be: a decline in military prowess and ineffective administration.\n"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 1424,
    "completion_tokens": 65,
    "total_tokens": 1489,
    "prompt_tokens_details": {
      "cached_tokens": 1388
    }
  }
}

On the cache hit, cached_tokens is 1388 and cache_write_tokens is omitted (no write). The cached content is reused, reducing input token cost.

Explicit caching requires a minimum of 1024 tokens in the cacheable content block. Shorter content returns a validation error.

Response Example (summary)

json

{
  "id": "chatcmpl-...",
  "usage": {
    "prompt_tokens": 4641,
    "completion_tokens": 1817,
    "prompt_tokens_details": {
      "cached_tokens": 4608
    }
  }
}

Usage Fields

Field	Description
`cached_tokens`	Number of tokens read from the cache (cache hit). When greater than zero, you benefit from cached content.
`cache_write_tokens`	Number of tokens written to the cache. Appears on the first request when establishing a new cache entry.

Some providers charge differently for cache writes vs. reads. Anthropic, for example, charges for cache writes but offers discounts on cache reads. Check provider pricing for details.