Prompt Caching

The Solution: Prompt Caching — Remember, Don't Repeat

Prompt caching is an API-level feature that stores the processed prefix of your prompt across requests so the model doesn't have to re-process the same tokens every time. Only the beginning is cached — never the middle or the end. Anthropic offers a 90% discount on cached input tokens, with a one-time 25% surcharge on the request that writes the cache. OpenAI caches automatically for prompts over 1,024 tokens at a 50% discount. The cache has a TTL (time-to-live, время жизни) of roughly 5 minutes by default, and that timer resets on every cache hit, so an actively used prefix effectively stays warm.

How it actually works under the hood

When a transformer processes your prompt, it computes intermediate attention states for every token — this is the KV-cache (key-value cache). Normally those states are thrown away after each request. Prompt caching keeps the states for the unchanged prefix in memory, so a repeat request skips the expensive "prefill" over those tokens and resumes from where the new content begins. That is why it cuts both cost and latency — the model is doing strictly less compute. The critical constraint: the cached portion must be byte-for-byte identical and sit at the very start. Change a single character near the front — a date, a user name, a reordered example — and the prefix no longer matches, so the cache misses and you pay full price.

When to use it (and a worked example)

Reach for caching whenever a large, stable block leads every request: a long system prompt, a set of few-shot examples, or reference documents in a RAG pipeline. Suppose a support bot prepends a 3,000-token system prompt plus 50 example tickets, then appends only the new user message. The first call is a cache write (you pay ~1.25x for the prefix). Every call in the next 5 minutes reuses that prefix at 10% of the price, while the short per-user suffix is billed normally. Over thousands of daily requests, the savings dominate. The main pitfall is structural: put everything static first, everything dynamic last. If the user query or a timestamp leaks into the front of the prompt, the cache never hits — and you've paid the write surcharge for nothing.

Where to Apply Prompt Caching

• RAG with stable system prompt: Cache the system instructions + retrieval guidelines. Only the user query and retrieved chunks change per request. Ideal for high-volume Q&A systems
• Few-shot classification: Cache 50-100 classification examples in the prefix. Each new input is appended at the end. Perfect for support ticket routing or content moderation
• Batch processing: Process thousands of documents with the same analysis prompt. Cache the instructions once, change only the document per request. Massive savings at scale
• Common Pitfall: Putting dynamic content before static content in your prompt. If the user query comes before the system prompt, every request has a different prefix and the cache never hits. Always structure: [stable prefix] + [dynamic suffix]

Try It Yourself!

Explore the interactive visualization below to see how caching affects cost, latency, and token usage in real scenarios.