Prompt Caching

Every request of a chat sends the same start again: the system prompt, the tools, and the earlier messages. You pay for those tokens on each call, and the model reads them again before it answers.

Prompt caching lets the provider keep that start. The next request with the same start reads it from the cache. It costs less, and the answer starts sooner.

chat() turns it on for you. You change nothing.

Give each conversation a key#

Pass the threadId of the request to chat(). chat() uses it as the cache key:

import {
  chat,
  chatParamsFromRequest,
  toServerSentEventsResponse,
} from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";

export async function POST(request: Request) {
  const { messages, threadId } = await chatParamsFromRequest(request);

  const stream = chat({
    adapter: openaiText("gpt-6.1-sol"),
    systemPrompts: ["You are a support agent for Acme."],
    messages,
    threadId,
  });

  return toServerSentEventsResponse(stream);
}

Nothing changes on the client. useChat already sends one threadId for each conversation.

The key is the first of these that you set:

1. promptCache.key.
2. The threadId or the conversationId that you pass to chat().
3. No key. Claude does not need one. OpenAI uses the key to route requests, so a stable threadId gives more cache hits.

A request reads the cache only up to the first token that differs. Text that changes on each call, such as the current time in the system prompt, ends the match there.

Turn it off, or keep it longer#

Set promptCache on chat():

To set your own key too, pass an object. Both fields are optional:

import { chat } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";

const stream = chat({
  adapter: openaiText("gpt-6.1-sol"),
  messages: [{ role: "user", content: "Where is my order?" }],
  promptCache: { retention: "long", key: "acme-support" },
});

The types come from @tanstack/ai: PromptCacheRetention, PromptCacheOptions, and ResolvedPromptCache.

Need a different cache for one model call only? A middleware can set it. See Change the prompt cache of a call.

Know what it costs on Claude#

On OpenAI and Mistral, a cache write is free and a cache read is cheaper. Keep the default.

Claude bills the cache write, and a read costs much less than normal input:

• With the 5-minute cache ('short'), a write costs a bit more than normal input. A start that you send twice already costs less than no cache.
• With the 1-hour cache ('long'), a write costs about twice the normal input. The start must be read more times before the cache saves money.

Two cases change the choice:

• A large prompt that you send one time. For example, a long document per call that you never send again. Nothing reads that cache, so the write is only extra cost. Use 'none'.
• Long pauses. The short Claude cache ends 5 minutes after the last read. If your users often reply later than that, use 'long'.

import { chat } from "@tanstack/ai";
import { anthropicText } from "@tanstack/ai-anthropic";

export function reviewContract(contract: string) {
  return chat({
    adapter: anthropicText("claude-opus-5-5"),
    messages: [{ role: "user", content: `Review this contract:\n\n${contract}` }],
    // Sent one time only: skip the cache write.
    promptCache: "none",
  });
}

Claude caches a start only when it is longer than a model minimum (512 to 4096 tokens). A shorter start costs nothing extra.

What each provider gets#

Other adapters send no cache fields. On gpt-5.6 and later, openaiText with 'none' sends prompt_cache_options: { mode: 'explicit' }, so OpenAI caches nothing.

Each adapter page has the details: Anthropic, Amazon Bedrock, OpenAI, OpenRouter, and Mistral.

Set the cache yourself#

Your own cache fields win. Then chat() adds no automatic fields of that kind to the request:

• Anthropic: cache_control in the metadata of a system prompt, a message part, or a tool, or modelOptions.cache_control.
• Bedrock: metadata.cachePoint on a system prompt, a message part, or a tool.
• OpenAI: modelOptions.prompt_cache_key or modelOptions.prompt_cache_retention.
• OpenRouter: modelOptions.sessionId.

Add tools without losing the cache#

The tools sit near the start of each request. Add a tool during a run, and the next call cannot read the cache from there. Some GPT and Claude models take the new tool inside the conversation instead, so the start stays the same. See Mid-Conversation Changes.

See the cache work#

usage.promptTokens is the full input. Two fields split it:

• usage.promptTokensDetails.cachedTokens: tokens read from the cache.
• usage.promptTokensDetails.cacheWriteTokens: tokens written to the cache. Claude reports it.

Log them in the onUsage hook. It runs one time for each model call:

import { chat } from "@tanstack/ai";
import type { ChatMiddleware } from "@tanstack/ai";
import { anthropicText } from "@tanstack/ai-anthropic";

const cacheLog: ChatMiddleware = {
  name: "cache-log",
  onUsage: (ctx, usage) => {
    const read = usage.promptTokensDetails?.cachedTokens ?? 0;
    console.log(`Call ${ctx.iteration}: ${read} of ${usage.promptTokens} tokens from the cache`);
  },
};

const stream = chat({
  adapter: anthropicText("claude-opus-5-5"),
  messages: [{ role: "user", content: "Hello!" }],
  threadId: "thread-1",
  middleware: [cacheLog],
});

From the second call of a conversation on, most of the input now comes from the cache.