Prompt Caching for Real-World LLM Apps

Method and confidence

This guide uses primary sources only: the current OpenAI prompt caching guide, the current Claude Platform prompt caching guide, and the relevant API reference pages linked from those docs. Where I describe implementation choices, I separate vendor facts from engineering judgment. I am not claiming a universal latency or cost improvement, because the result depends on prompt shape, cache hit rate, request volume, and model family.

The core thesis is simple: if your application keeps sending the same long prefix over and over, you are paying to reprocess the same text over and over. Prompt caching attacks that waste. It does not replace good prompt design, structured outputs, observability, or retrieval discipline. It just makes the stable part of the prompt cheaper to reuse.

Why repeated prefixes are the hidden tax

Most production LLM traffic is not a fresh blank page. It is a repeated instruction stack:

• System instructions
• Tool schemas
• Policy text
• Few-shot examples
• Retrieval scaffolds
• Tenant-specific framing

Only the last slice of the request usually changes. The problem is that a naive app sends the entire bundle every time, which forces the model service to re-encode the same prefix repeatedly.

That is expensive in two ways:

1. It increases input token cost.
2. It burns latency on work that could have been reused.

Once a prompt becomes large enough, the repeated prefix often matters more than the user message itself. That is especially true for agentic apps, support copilots, and RAG systems where the "real" prompt is the sum of instructions, schemas, and retrieved context. If you already care about schema discipline, the Structured Outputs guide shows why a stable contract matters; if you care about telemetry, OpenTelemetry GenAI observability is the companion piece.

What prompt caching actually does

The OpenAI guide defines prompt caching as a way to reduce latency and cost for long prompts. It says caching is available for prompts containing 1024 tokens or more, and it exposes cache-read and cache-write usage fields so you can measure what is actually happening. The current docs also describe prompt_cache_key for routing related requests to the same cache and prompt_cache_retention on older model families.

Anthropic frames the same idea with cache_control blocks and explicit cache breakpoints. Its docs describe automatic caching and manual breakpoints, plus 5-minute and 1-hour TTLs. The important distinction is not philosophical. It is operational: OpenAI emphasizes request-level routing and retention policy, while Anthropic emphasizes block-level annotations and cache performance fields.

This screenshot proves two things. First, the OpenAI guide is current and public. Second, the product positioning is explicit: prompt caching exists to reduce latency and cost for long prompts, not to change the model's output semantics.

OpenAI also says prompt caches are not shared across organizations and that prompt caching does not change how output tokens are generated. That privacy boundary matters if you run multi-tenant systems. Cache reuse can be an efficiency feature without becoming a data-leak feature.

This second screenshot proves the cross-vendor shape of the feature. Anthropic is solving the same class of problem, but it exposes the control surface differently: cacheable blocks, breakpoints, and cache performance accounting.

Design the prefix so it can be reused

The biggest mistake teams make is treating prompt caching as a magical toggle. It is not. You have to design the prompt so the stable prefix is actually stable.

The prefix is the part that should stay invariant across requests:

• Role instructions
• Safety policy
• Tool definitions
• Schema definitions
• Tenant policy
• Few-shot examples

The tail is the part that should change:

• User message
• Current query
• Retrieved passages
• Request-specific metadata
• Short-lived session state

If you keep moving instructions between the prefix and the tail, you destroy the cache line you were trying to preserve.

What breaks reuse

The current docs give several concrete examples of cache invalidation. On OpenAI, changing the request shape, the cache key strategy, or the retention policy can affect matching. On Anthropic, changes to tool definitions, web-search toggles, citations, speed settings, tool choice, images, thinking parameters, or effort settings can invalidate cached blocks at different scopes.

That is the practical lesson: treat your prompt contract like code.

If you have a stable support assistant, keep the assistant policy versioned. If you have tool schemas, version those too. If you have a retrieval system, separate the retrieval scaffold from the live query. If you have per-tenant variations, keep the tenant boundary visible in the cache key or prompt annotation instead of smearing tenant data through the entire prefix.

A practical implementation pattern

The safest pattern is to compose the prompt in three layers:

1. Stable prefix
2. Cached routing identity
3. Dynamic tail

Here is a minimal TypeScript sketch:

1import { createHash } from "node:crypto";
2
3function hash(parts: string[]) {
4  return createHash("sha256").update(parts.join("\n")).digest("hex");
5}
6
7function buildPromptBundle(input: {
8  tenantId: string;
9  policyVersion: string;
10  toolSchemaVersion: string;
11  systemPrompt: string;
12  userMessage: string;
13}) {
14  const cacheKey = hash([
15    input.tenantId,
16    input.policyVersion,
17    input.toolSchemaVersion,
18  ]);
19
20  const stablePrefix = [
21    input.systemPrompt,
22    `policy_version=${input.policyVersion}`,
23    `tool_schema_version=${input.toolSchemaVersion}`,
24  ].join("\n\n");
25
26  return {
27    cacheKey,
28    stablePrefix,
29    userTail: input.userMessage,
30  };
31}

The point of the hash is not cryptographic security. It is routing discipline. If the stable prefix changes, the key changes. If the key changes every request, the cache is basically dead.

For OpenAI-style requests, the docs now expose prompt_cache_key and usage fields that tell you how much was read from cache and how much was written. For GPT-5.6 and later model families, the docs say the key is required for more reliable matching. That makes the cache key part of the contract, not an optional hint.

1const bundle = buildPromptBundle({
2  tenantId: "tenant:acme",
3  policyVersion: "2026-07",
4  toolSchemaVersion: "3",
5  systemPrompt: "You are a concise support assistant.",
6  userMessage: "Why did invoice 1827 fail?",
7});
8
9const response = await openai.responses.create({
10  model: "gpt-5.6",
11  prompt_cache_key: bundle.cacheKey,
12  prompt_cache_retention: "24h",
13  input: [
14    { role: "system", content: bundle.stablePrefix },
15    { role: "user", content: bundle.userTail },
16  ],
17});
18
19const cached = response.usage?.input_tokens_details?.cached_tokens ?? 0;
20const written = response.usage?.input_tokens_details?.cache_write_tokens ?? 0;
21console.log({ cached, written });

That example is intentionally boring. Good cache usage should be boring. The interesting part is the surrounding policy: keep the stable prefix versioned, inspect cache reads and writes, and compare the economics against the uncached path before you declare victory.

For Anthropic-style requests, the docs center cache_control and explicit breakpoints. The mental model is the same: annotate the stable blocks, keep the tail small, and use the token accounting fields to see whether reuse is real.

OpenAI and Anthropic solve the same problem differently

This table is the shortest honest summary I can give. The providers are not identical, but the operational lesson is the same: stabilize the prefix, make the dynamic tail as small as possible, and instrument the result.

A cache-friendly request path

This diagram is the architecture I would actually want in production. The user request should pass through prompt assembly, then into a cache-aware routing layer, then into the model. The cache should be visible in logs and metrics, not hidden inside a mystery library call.

If the cache hit rate is low, the answer is usually not "buy more GPUs." It is "why are we changing the prefix so often?" In other words, this is as much a prompt-design problem as a systems problem.

Where prompt caching pays off fastest

Prompt caching is most useful when the request shape is repetitive and the prefix is long:

• Support and success copilots
• RAG systems with fixed policy scaffolding
• Multi-tenant assistants with one policy stack per tenant
• Tool-using agents with stable tool schemas
• Internal workflow bots that reuse the same instructions all day

It is less useful when every request is unique, the prompt is tiny, or the model configuration changes constantly. If the repeated prefix is only a few dozen tokens, the operational complexity may not be worth it. If your real bottleneck is generation quality, prompt caching will not fix that either.

Do not ignore the other bottlenecks

Cache hits only matter if the rest of the stack is disciplined:

• vLLM PagedAttention and Continuous Batching explains why serving efficiency still depends on memory and scheduling.
• MCP tool-server threat modeling explains why tool boundaries need their own security review.
• Secure AI Containers with SBOMs and Provenance covers the release integrity side of the same production system.

Prompt caching is one optimization in a larger stack. If the prompt is sloppy, the cache hit rate will be sloppy. If the observability is weak, you will not know whether it helped. If the prompt contract changes every deploy, you will churn the cache and blame the model for your own instability.

Production checklist

Before you ship prompt caching, I would require all of the following:

1. A versioned stable prefix with a clearly documented owner.
2. A tenant-safe cache key strategy.
3. Logging for cache reads, cache writes, and total input tokens.
4. A change-management rule for system prompts, tool schemas, and retrieval scaffolds.
5. A canary path that compares cached and uncached behavior.
6. A rollback path if cache behavior changes after a model upgrade.
7. A privacy review for any cross-request state that touches cached prefixes.

If you cannot explain why a prompt is in the stable prefix, it probably should not be there.

FAQ

What exactly is cached?

The stable prefix is cached, not the whole request. On OpenAI, the docs describe cached prefixes and usage fields for reads and writes. On Anthropic, cached blocks are marked with cache_control or chosen through explicit breakpoints.

Is prompt caching only useful for huge prompts?

It becomes most visible on long prompts, because the repeated prefix is large enough to matter. But the real threshold is not a marketing number; it is whether the repeated prefix is expensive enough to justify the added prompt discipline.

Does prompt caching change the model output?

No, not by itself. OpenAI says caching does not change how output tokens are generated. The model still computes a fresh answer from the cached prefix, so nondeterministic requests can still vary.

Should I put tenant IDs in the cache key?

Usually yes, if the prefix is tenant-specific or the policy text varies per tenant. The key should preserve isolation and routing correctness. The actual format is up to you, but the boundary should be explicit.

What is the biggest cache killer?

Unnecessary prefix churn. If you keep changing the system prompt, tool definitions, or prompt-shaping settings, you keep invalidating the reuse you wanted.

How do I know the cache is helping?

Measure it. On OpenAI, inspect cached_tokens and cache_write_tokens. On Anthropic, inspect cache_creation_input_tokens, cache_read_input_tokens, and the remaining input_tokens. Then compare the economics to the uncached path.

Should I use prompt caching instead of retrieval?

No. They solve different problems. Retrieval changes what evidence the model sees; caching makes repeated evidence cheaper to reuse. In a serious app, you often want both.

Sources and related reading

Primary sources used for this guide:

• OpenAI Prompt Caching guide
• OpenAI Responses API reference for prompt_cache_key
• Claude Platform Prompt Caching docs
• Claude Platform docs on cache performance

If you want the adjacent systems pieces, the most relevant follow-ups on this site are the Structured Outputs guide, the OpenTelemetry GenAI observability guide, and the vLLM PagedAttention and Continuous Batching article.

The practical rule is unchanged: make the prefix stable, keep the tail small, and verify the economics with your own telemetry before you trust the headline.