Prompt caching

Long prefixes you send repeatedly (system prompts, documents, tool definitions) can hit the cache, cutting input cost and latency significantly.

How it works

In multi-turn chat, agents, RAG and similar workloads, every request starts with a large block of identical content. With prompt caching, that block is "written" to the cache the first time, and later requests "read" it directly; the cached part is billed far below the regular input price.

Claude: mark cache_control manually

Add "cache_control": {"type": "ephemeral"} at the end of the content block you want cached. Everything up to that point is cached (5-minute lifetime by default, refreshed on every hit):

Python

import anthropic

client = anthropic.Anthropic(api_key="sk-xxx", base_url="https://api.onefast.ai")
long_doc = open("manual.txt").read()  # a long reference document

for question in ["What is chapter 1 about?", "Summarize chapter 3"]:
    resp = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        system=[{"type": "text", "text": long_doc, "cache_control": {"type": "ephemeral"}}],
        messages=[{"role": "user", "content": question}],
    )
    u = resp.usage
    print(u.cache_creation_input_tokens, u.cache_read_input_tokens, u.input_tokens)

On the first request, cache_creation_input_tokens is the amount written to the cache; on later requests with the same prefix, cache_read_input_tokens is the amount hit.

• Minimum cacheable length: 1024 tokens for most Claude models; anything shorter isn't cached.
• You can set up to 4 cache_control breakpoints (e.g. one each for tool definitions, system prompt and message history).
• Claude Code uses caching automatically; no setup needed.

GPT / Gemini / DeepSeek: automatic caching

These models need no parameters: when the prefix exceeds 1024 tokens, the upstream tries to cache it automatically. Hits show up in usage:

• OpenAI format: usage.prompt_tokens_details.cached_tokens
• Gemini native: usageMetadata.cachedContentTokenCount

The hit rate of automatic caching is determined by the upstream and depends on route, request interval, load and so on — a hit is not guaranteed every time; the hit rate rises over several consecutive requests with the same prefix. Hits are billed at the "cache read" price in the model catalog.

Improving the hit rate

• Put unchanging content first (system prompt → tool definitions → reference material) and changing content (the user's question) last.
• The prefix must be byte-for-byte identical: keep timestamps and random IDs out of it.
• Use the same key and group within a session where possible; the gateway tries to route identical prefixes to the same route.