---
name: "token_economy"
description: "Control LLM cost and latency: prompt caching, context discipline, model routing, batching, and measuring what you spend."
---

# Token Economy

For an agent, tokens are money and latency is user patience. Both
compound: a 5-minute cron job that re-sends a 60K system prompt every
turn is burning budget all day. Spend deliberately.

## The big levers (in order of impact)

### 1. Prompt caching — stop re-paying for the same prefix

Multi-turn agents re-send the system prompt and tool schemas every turn.
Cache the stable prefix instead:

- **Anthropic:** mark stable blocks with
  `cache_control: {"type": "ephemeral"}` on the system message and tool
  definitions. Cache reads bill at **10% of input price** (90% off).
  Cache lives ~5 minutes of inactivity; the initial write costs a small
  premium once.
- **OpenAI:** automatic for prompts ≥ 1024 tokens sharing a common
  prefix — no code changes, ~50% off cached input. Just keep the prefix
  stable and in the same order.
- Rule: anything identical across turns goes *before* the cache
  breakpoint; anything that changes goes after. A timestamp or run-id at
  the top of the system prompt silently invalidates the whole cache.

### 2. Context discipline — the context window is not a junk drawer

- Summarize aggressively. A rolling summary of completed work beats 50
  raw tool outputs nobody will re-read.
- Drop what's served its purpose: full file contents after the edit is
  applied, search results after the answer is extracted.
- Prefer references over inlining: "see `design.md` section 3" costs
  nothing; pasting the section costs tokens every turn until compaction.
- Watch for the same content entering twice (a tool result echoed into
  a file that's then re-read). Dedupe at the source.

### 3. Model routing — match the model to the job

- Cheap/fast models for classification, extraction, formatting,
  first-pass triage. Frontier models for reasoning, planning, and
  anything the user will see.
- A common shape: small model runs the loop, big model reviews the
  final output. You get most of the quality at a fraction of the cost.
- Don't route on vibes — route on measured quality. Spot-check the cheap
  model's output on real tasks before trusting it with the job.

### 4. Batching and parallelism

- Independent tool calls go in one block, not five sequential turns.
  Each turn re-sends context; batching turns is batching cost.
- But: don't parallelize to hide latency you should be eliminating.
  Five parallel calls that each fetch 10MB are a context problem, not a
  latency win.

### 5. Kill the retry loops

- A step that fails and retries 10 times with full context each time is
  a cost incident. Cap retries (3–5), back off, and escalate.
- Timeouts on every model call. A hung call that retries forever is
  unbounded spend.

## Measure it

You can't optimize what you don't track:

- Log per run: input tokens, output tokens, cache hit rate, model,
  duration. One line per run is enough.
- Watch the ratios: if input is >90% of your bill, caching and context
  discipline are your levers. If output dominates, your responses are too
  long — tighten the output contract.
- Set a budget per job and alert at 80%. A cron job with no budget is a
  blank check.

## Operating Rules

1. Stable prefixes go first, volatile content last — cache breakpoints
   only work if you respect the ordering.
2. Never put timestamps, UUIDs, or run counters inside cached blocks.
3. Measure before and after every optimization. "Felt faster" is not data.
4. Latency matters too: stream to the user when they'll wait, batch when
   they won't notice.
5. The cheapest token is the one you never send. Delete before you
   compress, compress before you cache.