token_economy v1.0.0

name: "token_economy" description: "Control LLM cost and latency: prompt caching, context discipline, model routing, batching, and measuring what you spend."

Token Economy

For an agent, tokens are money and latency is user patience. Both compound: a 5-minute cron job that re-sends a 60K system prompt every turn is burning budget all day. Spend deliberately.

The big levers (in order of impact)

1. Prompt caching — stop re-paying for the same prefix

Multi-turn agents re-send the system prompt and tool schemas every turn. Cache the stable prefix instead:

• Anthropic: mark stable blocks with cache_control: {"type": "ephemeral"} on the system message and tool definitions. Cache reads bill at 10% of input price (90% off). Cache lives ~5 minutes of inactivity; the initial write costs a small premium once.
• OpenAI: automatic for prompts ≥ 1024 tokens sharing a common prefix — no code changes, ~50% off cached input. Just keep the prefix stable and in the same order.
• Rule: anything identical across turns goes before the cache breakpoint; anything that changes goes after. A timestamp or run-id at the top of the system prompt silently invalidates the whole cache.

2. Context discipline — the context window is not a junk drawer

• Summarize aggressively. A rolling summary of completed work beats 50 raw tool outputs nobody will re-read.
• Drop what's served its purpose: full file contents after the edit is applied, search results after the answer is extracted.
• Prefer references over inlining: "see design.md section 3" costs nothing; pasting the section costs tokens every turn until compaction.
• Watch for the same content entering twice (a tool result echoed into a file that's then re-read). Dedupe at the source.

3. Model routing — match the model to the job

• Cheap/fast models for classification, extraction, formatting, first-pass triage. Frontier models for reasoning, planning, and anything the user will see.
• A common shape: small model runs the loop, big model reviews the final output. You get most of the quality at a fraction of the cost.
• Don't route on vibes — route on measured quality. Spot-check the cheap model's output on real tasks before trusting it with the job.

4. Batching and parallelism

• Independent tool calls go in one block, not five sequential turns. Each turn re-sends context; batching turns is batching cost.
• But: don't parallelize to hide latency you should be eliminating. Five parallel calls that each fetch 10MB are a context problem, not a latency win.

5. Kill the retry loops

• A step that fails and retries 10 times with full context each time is a cost incident. Cap retries (3–5), back off, and escalate.
• Timeouts on every model call. A hung call that retries forever is unbounded spend.

Measure it

You can't optimize what you don't track:

• Log per run: input tokens, output tokens, cache hit rate, model, duration. One line per run is enough.
• Watch the ratios: if input is >90% of your bill, caching and context discipline are your levers. If output dominates, your responses are too long — tighten the output contract.
• Set a budget per job and alert at 80%. A cron job with no budget is a blank check.

Operating Rules

1. Stable prefixes go first, volatile content last — cache breakpoints only work if you respect the ordering.
2. Never put timestamps, UUIDs, or run counters inside cached blocks.
3. Measure before and after every optimization. "Felt faster" is not data.
4. Latency matters too: stream to the user when they'll wait, batch when they won't notice.
5. The cheapest token is the one you never send. Delete before you compress, compress before you cache.