prompt_caching_strategy v1.0.0

name: "prompt_caching_strategy" description: "Cut LLM input costs with prompt caching: design cacheable prefix layouts, place cache breakpoints, pick TTLs, measure hit rates, and run break-even cost math across Anthropic, OpenAI, and Gemini. Triggers on LLM cost optimization, caching configuration, or why a cache never hits." metadata: { "includeInPrompt": true }

Prompt Caching Strategy

Purpose

Turn a repeated LLM prompt into a cache-friendly layout so identical prefixes get served cheap instead of re-encoded every call. Covers breakpoint placement (Anthropic), automatic caching behavior (OpenAI), explicit context caching (Gemini), hit-rate measurement, and the break-even math that decides whether caching pays off.

Workflow

1. Inventory the request shape. List everything sent on every call, in render order: tool definitions, system prompt, few-shot examples, retrieved documents, conversation history, current user message. Note which parts change call to call.

1. Pick the provider mechanism. See provider specs.

• Anthropic: explicit cache_control breakpoints (up to 4 per request).
• OpenAI: automatic, no configuration needed, prefix >= 1,024 tokens.
• Gemini: explicit CachedContent API for long-lived prefixes.

1. Order the request stable-first. Put stable content at the top, volatile content at the bottom. The layout that hits:

   [tool definitions]      <- cache (Anthropic: breakpoint on last tool block)
   [system prompt]         <- cache (Anthropic: breakpoint on last system block)
   [few-shot examples]     <- cache
   [retrieved documents]   <- cache only if reused across calls
   [conversation history]  <- cache up to the last stable turn
   [current user message]  <- never cache; changes every time
   

A single byte difference in the prefix invalidates everything after it. Timestamp, request id, or session token placed early in the prompt silently kills every hit.

1. Set TTL by traffic rhythm. 5-minute (Anthropic default, OpenAI automatic) fits chat/agent loops with back-to-back turns. 1-hour (Anthropic {"type":"ephemeral","ttl":"1h"}, Gemini CachedContent) fits bursty workloads with 20-60 minute gaps. The 1-hour write costs 2x, so it needs 3+ reads to break even.

1. Prove it works from usage fields, not vibes.

• Anthropic: cache_read_input_tokens > 0 on the second request, and cache_creation_input_tokens = 0 after the first write.
• OpenAI: usage.prompt_tokens_details.cached_tokens > 0.
• If reads stay at 0, walk the cache-killer checklist before changing anything else.

1. Run the cost math. Use bin/cache_math.py to compute break-even and savings from real usage numbers. Rule of thumb: cache only prefixes reused at least twice within the TTL window.

Output Contract

A caching recommendation states: provider, what content sits at each cacheable layer, where breakpoints go (Anthropic), the TTL and why, the break-even point in requests, and how the agent will verify hits from usage fields. If caching does not pay off for a workload, say so and recommend skipping it.

Operating Rules

1. Verify every factual claim (prices, minimums, TTLs) against current provider docs or pricing pages. Never quote a multiplier from memory in a client-facing recommendation.
2. Below-minimum prefixes silently no-op on Anthropic. Check the minimum for the exact model before assuming a write happened.
3. One write plus one read loses money on Anthropic (1.25 + 0.1 > 1). Never recommend caching a prompt that runs twice or fewer per TTL window.
4. Never recommend parallel fan-out requests expecting shared cache hits: the first response must start streaming before the entry exists.
5. Never let a session token, timestamp, or rotating key sit ahead of a breakpoint. Volatile content goes after the last cache marker, always.