context_window_budgeting v1.0.0

name: context_window_budgeting description: Budget an LLM's context window for long agent runs. Decides what stays in context, what gets summarized, cleared, or retrieved on demand; sets compaction checkpoints; measures token cost per task. Use when a session is long-running, token costs are climbing, or the agent is losing track mid-task.

Context Window Budgeting

Purpose

Treat the context window as a budget, not a box. Every token in context is rented per turn: it costs money, latency, and attention. This skill gives an agent a repeatable way to decide what earns its rent, what gets evicted, and how to measure the bill per task.

Workflow

1. Inventory. List what is currently in context and estimate its token size: system prompt, tool definitions, project conventions, retrieved references, conversation history, tool outputs. For a quick estimate, use bin/context_cost.py estimate <file> (counts words, applies a 1.3 tokens-per-word heuristic for English prose).

1. Zone it. Assign every item to a zone:

• Hot (always keep): goal, constraints, working notes the agent rewrites each turn. Small by design.
• Warm (keep recent N): last few conversation turns, most recent tool outputs. N is small, like 3 to 5.
• Cold (summarize or clear): older turns, processed tool outputs, resolved errors. Summarize into a scratch note; clear raw tool results once their facts are extracted.
• Frozen (never in context): full files, whole histories, large data dumps. Keep as paths or IDs; retrieve just-in-time via tools.

1. Order for recall and cache. Put instructions and goal first, the current question last (models recall the ends better than the middle, per Liu et al.). Keep the stable prefix (system prompt, tool defs, conventions) identical across calls so prompt caching discounts apply; a reshuffled prefix kills cache hits.

1. Set compaction checkpoints. Do not wait for the window to fill. Plan compaction at fixed points:

• ~55% of capacity: routine compaction, summarize older turns.
• ~75%: aggressive compaction, clear all non-recent tool results.
• ~90%: emergency compaction, keep only the working note + last 3 turns.
• Task boundaries are the best compaction moments: compact between subtasks, never mid-thought.
• See references/compaction_playbook.md for the keep/discard table and a summary template.

1. Measure. After the task, run bin/context_cost.py report against the session's token log (or a manually recorded estimate) to get cost per task. Track: total tokens, cost, and what percentage was hot-zone material. If cold-zone tokens dominate, the next run needs tighter clearing.

Worked example

An agent with a 200K-token window on a 2-hour research task:

| Zone | What | Est. tokens | |---|---|---| | Hot | goal, constraints, running note | 2,000 | | Warm | last 4 turns + latest tool output | 8,000 | | Cold | compaction summary of first 90 min | 3,000 | | Frozen | 40 source documents (paths only, retrieved on demand) | 200 |

Keeping the documents in context would have been ~120,000 tokens per turn. Retrieving on demand: ~5,000. Same information, roughly 24x cheaper, and the model reads the retrieved chunk at the high-recall tail instead of buried in the middle.

Output Contract

When asked to budget context for a task, return:

• Budget table: each item in context, its zone, and estimated tokens.
• Cut list: what moves to cold/frozen and what is cleared outright.
• Checkpoint plan: when compaction fires (capacity thresholds or task boundaries).
• Cost estimate: tokens and cost per expected turn, from bin/context_cost.py.

Operating Rules

1. Never invent token counts, savings percentages, or model limits. Estimate and label estimates as estimates; cite a measured source or say you do not know.
2. Selection beats stuffing. If the user asks about one invoice, load one invoice, not the ledger.
3. Summaries replace transcripts; identifiers replace data. Keep file paths, record IDs, and query text. Fetch the payload only when the current step needs it.
4. Compact early, compact often. Summarizing 10 turns with headroom beats emergency compaction that loses the one detail that mattered.
5. Instructions from early in the conversation can be lost by compaction. Re-state critical constraints after each compaction, or pin them in the hot zone.
6. Do not reorder the stable prompt prefix across turns. Cache-hit order: system prompt, tool definitions, project conventions, retrieved references, user message.
7. Measure per task, not per turn. A single cheap turn that sets up 50 expensive ones is not cheap.