Cost and latency
Prompt caching
Reduce AI model costs with implicit and explicit prompt caching
Use prompt caching to save on inference costs where the provider and model support it. Inworld Inference supports implicit and explicit caching.
Implicit Caching
Implicit caching works automatically when the model provider supports it. You do not need to configure it, but the provider's rules apply.
Provider stickiness: A cache lives with one provider, so only requests sent to that provider can read it. A model ID with a provider prefix (for example, google-ai-studio/gemini-2.5-flash) always goes to that provider. A model name without a prefix (for example, deepseek-v4-flash) is different: the router picks a provider for each request, trying the lowest-latency provider first and moving to the next one if it fails. Its latency ranking is refreshed about once a minute, so two requests in one conversation can reach different providers, and the second does not read the first one's cache. To keep a conversation on one provider, use the prefixed ID, or set provider.order with allow_fallbacks: false (provider routing). If you call a router that splits traffic across variants, send the same user on every request of a conversation to keep it on one variant (sticky routing). A request that falls back to another model, or for which model: "auto" picks a different model, goes to that model's provider and does not read the earlier cache.
Many models cache implicitly, with no change to your request. Some models also support explicit caching, described below: a request with cache_control uses an explicit cache, and a request without it can still get the provider's implicit caching. Minimum token counts and TTL behavior are the provider's and vary by model; check the provider's documentation for details.
You can also send OpenAI's prompt_cache_key and prompt_cache_retention fields. Inworld passes them only to models that take them; other models do not receive them, without an error. See OpenAI's prompt caching guide for what they do and the values each model accepts.
To see which models support prompt caching and what cached tokens cost, browse inworld.ai/models or call the List models API: check spec.capabilities.promptCaching for support, and pricing.promptCacheReadToken and pricing.promptCacheWriteToken for the per-token rates. How caching is billed gives the counting rule.
Explicit Caching
Explicit caching works on some models. It gives you control over what gets cached and for how long. A request that writes an explicit cache reports cache_write_tokens in usage (below); on a model without explicit caching, no explicit cache is written, and the provider's implicit caching, if any, still applies. Use it on /v1/chat/completions (shown below) or on the Anthropic-compatible Messages API.
Unified Protocol
Every model with explicit caching uses the same protocol: a cache_control object in each message content part, including a configurable ttl (time to live).
Cache Control in Messages
Add cache_control to text parts within multipart message content. Reserve it for large bodies of text such as character cards, CSV data, RAG context, or book chapters. The cache covers everything up to and including the part that carries the breakpoint.
{
"messages": [
{
"role": "system",
"content": [
{
"type": "text",
"text": "You are a historian studying the fall of the Roman Empire. Below is an extensive reference book:"
},
{
"type": "text",
"text": "HUGE TEXT BODY HERE",
"cache_control": {
"type": "ephemeral",
"ttl": "1h"
}
}
]
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "What triggered the collapse?"
}
]
}
]
}TTL Management
- Configurable TTL: Set
ttlin thecache_controlobject (e.g.,"ttl": "5m","ttl": "1h"). The accepted values depend on the model: some accept only"5m"and"1h", and others accept other durations, such as"30m". Withoutttl, the provider's default applies. - Update at any time: You can change the TTL on later requests. On Google models, a request with a different
ttland the same cached content changes the lifetime of the existing cache: it does not create a new cache or a new cache write. On Anthropic models, see Anthropic's rules for cache durations. - Automatic prolongation: Anthropic refreshes the cache lifetime each time a request reads it. On Google models, when 10% or less of the TTL remains and a request with
ttlset reads the cache, Inworld Inference extends the cache by the full TTL to keep it warm.
User Message Example
{
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Based on the book text below:"
},
{
"type": "text",
"text": "HUGE TEXT BODY HERE",
"cache_control": {
"type": "ephemeral",
"ttl": "1h"
}
},
{
"type": "text",
"text": "List all main characters mentioned in the text above."
}
]
}
]
}Tools, whole messages and the whole request
cache_control also works in these places on /v1/chat/completions:
- On a tool definition: add
cache_controlnext totypeandfunctionin atoolsentry to cache the tool definitions up to and including that tool. Anthropic caches tools first, then the system prompt, then messages, so put the breakpoint on the last tool. - On a message with string content: when
contentis a plain string, putcache_controlon the message itself to mark the whole message. For multipart content, put it on a content part instead. - At the top level of the request (some models only): one
cache_controlobject next tomessages. The provider puts the breakpoint on the last cacheable block and moves it forward as the conversation grows, so you don't mark individual parts. Other models ignore it.
On Google models, the last breakpoint in the request (on a tool, a message or a content part) sets what is cached, and the cache always includes the tool definitions. Google doesn't accept a system prompt next to a cache, so keep system messages before the breakpoint: a system message after it is not sent. Put at least one message after the breakpoint: when the breakpoint covers every message, the request runs without the explicit cache.
One cache per model on Google
On Google models, Inworld keeps one explicit cache per workspace and model. A request whose cached content differs from the current cache (for example, a different character card) creates a new cache and replaces the old one, so two prompts that alternate keep rewriting the cache. To keep separate caches, add a cache_id string to cache_control, such as {"type": "ephemeral", "ttl": "1h", "cache_id": "character-42"}. Each cache_id gets its own cache. Anthropic ignores cache_id.
Inspecting Cache Usage
To see how much caching saved on each generation, check the prompt_tokens_details object in the usage response.
Request and Response Samples
The following samples were verified against the Inworld Inference API with a Google model. The first request establishes a cache; the second request hits the cache.
Request 1 — Cache write (establishing cache)
curl -X POST "https://api.inworld.ai/v1/chat/completions" \
-H "Authorization: Basic <your-api-key>" \
-H "Content-Type: application/json" \
-d '{
"model": "google-ai-studio/gemini-2.5-flash",
"messages": [
{
"role": "system",
"content": [
{
"type": "text",
"text": "You are a helpful historian. Below is reference material:"
},
{
"type": "text",
"text": "<large text body of 1024+ tokens>",
"cache_control": {"type": "ephemeral", "ttl": "5m"}
}
]
},
{"role": "user", "content": "What year did the Western Roman Empire fall? Reply in one short sentence."}
],
"max_tokens": 50
}'Response 1 — Cache write
{
"id": "chatcmpl-1772169766796",
"model": "google-ai-studio/gemini-2.5-flash",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "The Western Roman Empire fell in 476 AD.\n"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 1402,
"completion_tokens": 13,
"total_tokens": 1415,
"prompt_tokens_details": {
"cache_write_tokens": 1388,
"cached_tokens": 1388
}
}
}On Google models, Inworld first creates the cache from the marked content and then sends the request, so the model reads that content from the new cache. The first response therefore reports the same tokens twice: as cache_write_tokens (written to the cache) and as cached_tokens (read from it). On Anthropic models, the first request reports only cache_write_tokens, with cached_tokens at 0.
Request 2 — Cache hit (same conversation, follow-up question)
Send the same system prompt and cached content, plus the previous exchange and a new user message:
curl -X POST "https://api.inworld.ai/v1/chat/completions" \
-H "Authorization: Basic <your-api-key>" \
-H "Content-Type: application/json" \
-d '{
"model": "google-ai-studio/gemini-2.5-flash",
"messages": [
{
"role": "system",
"content": [
{"type": "text", "text": "You are a helpful historian. Below is reference material:"},
{
"type": "text",
"text": "<same large text body as request 1>",
"cache_control": {"type": "ephemeral", "ttl": "5m"}
}
]
},
{"role": "user", "content": "What year did the Western Roman Empire fall? Reply in one short sentence."},
{"role": "assistant", "content": "The Western Roman Empire fell in 476 AD.\n"},
{"role": "user", "content": "Name two factors that contributed to its fall."}
],
"max_tokens": 80
}'Response 2 — Cache hit
{
"id": "chatcmpl-1772169775239",
"model": "google-ai-studio/gemini-2.5-flash",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "While the provided text doesn't explicitly detail the reasons for the fall of the Western Roman Empire, it does highlight factors key to its rise and endurance... Drawing inferences from that, two factors could be: a decline in military prowess and ineffective administration.\n"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 1424,
"completion_tokens": 65,
"total_tokens": 1489,
"prompt_tokens_details": {
"cached_tokens": 1388
}
}
}On the cache hit, cached_tokens is 1388 and cache_write_tokens is omitted (no write). The cached content is reused, reducing input token cost.
Providers set a minimum size for a cached block, often 1,024 tokens or more, and it varies by model. These are the provider's limits, not Inworld's; check the provider's documentation for the model you use. Below the minimum, Google models return a 400 error ("The cached block is below the model's minimum cacheable size."), while Anthropic processes the request without caching and returns no error.
Response Example (summary)
{
"id": "chatcmpl-...",
"usage": {
"prompt_tokens": 4641,
"completion_tokens": 1817,
"prompt_tokens_details": {
"cached_tokens": 4608
}
}
}Usage Fields
| Field | Description |
|---|---|
cached_tokens | Number of tokens read from the cache (cache hit). Always present; 0 when nothing was read. When greater than zero, you benefit from cached content. |
cache_write_tokens | Number of tokens written to the cache. Appears only when a request writes a new cache entry. On Google models, the request that writes the cache also reads it, so cached_tokens is set too. |
On the Messages API (/v1/messages), the same counts appear as cache_read_input_tokens and cache_creation_input_tokens, and input_tokens is the rest of the prompt: it does not include either cache count, unlike prompt_tokens. Streamed chat completions carry the same prompt_tokens_details in the chunk that includes usage (Streaming).
How caching is billed
A request's prompt tokens are billed in three parts, each from the usage the response returns:
cached_tokensatpricing.promptCacheReadToken.cache_write_tokensatpricing.promptCacheWriteToken.- The rest of
prompt_tokensat the model's input rate. On Google models, Inworld writes the cache in a separate call, soprompt_tokensdoes not include the write: the rest isprompt_tokensminuscached_tokens(14 tokens in Response 1 above). On other providers, it isprompt_tokensminuscached_tokensandcache_write_tokens.
Get each model's rates from the List models API. They differ by provider: Anthropic, for example, charges more than the input rate for cache writes and less for cache reads.
Inworld currently bills nothing for how long an explicit cache is kept. On Google models, where Inworld creates the cache, you pay for the write and read tokens above and not for cache storage time, whatever the ttl.