llm_fallback_chains v1.0.0

name: llm_fallback_chains description: Designs resilient LLM model routing: tiered fallback chains, routing rules, per-request retry budgets, circuit breakers, and degraded-mode prompts. Use when an app calls one or more LLM providers and needs to survive rate limits, timeouts, and outages without hanging, silently degrading, or double-firing side effects.

LLM Fallback Chains

Purpose

Keep an LLM-powered feature working when providers misbehave. One provider down, rate-limiting, or timing out should move traffic to the next tier in milliseconds and tell you it happened, never hang the user and never pretend a weaker answer is the normal one.

This skill covers routing (which model serves which request), the chain itself (retries, timeouts, one shared deadline), circuit breaking, and what the product does when every model is unavailable. For client-side rate-limit pacing alone, see api_rate_limit_design.

Workflow

1. Name logical routes, not provider strings. Route code through logical tier names like smart, fast, budget. Reordering or swapping providers then becomes a config change, never a code change.

1. Pick a routing rule. Simplest first:

• Rule-based: classify the task (extraction, chat, reasoning) and assign it to a tier. Cheapest option, zero added latency.
• Complexity heuristic: short structured request goes to the budget tier; long open-ended request goes to the strong tier.
• Cascade: try the budget tier first; escalate only when confidence is low or the output fails validation.
• Managed: use a gateway (LiteLLM proxy, OpenRouter, or similar) for routing, fallback, and observability out of the box when you want less custom code. See references/routing-and-budgets.md for how to choose.

1. Build the chain short and ordered. Primary, then one or two known-good alternates, then a hard stop. Long chains add latency without adding reliability and hide which provider actually served the request. A different provider at the same tier beats a weaker model at the same provider when the problem is an outage.

1. Retry the tier, then fail over. Within one tier: retry transient failures (429, 5xx, timeouts, network blips) with exponential backoff and jitter, bounded attempts. Then move to the next tier. Never retry 4xx caller errors or auth failures. Never retry a step whose first attempt already acted (sent email, wrote a record, charged a card): retry the model call, not the side effects.

1. Set one deadline per request, not N timeouts. Give each request a single wall-clock budget. Every attempt's timeout is the smaller of its per-attempt timeout and the remaining budget, and an attempt that cannot fit in what is left never starts. Withhold a fallback reserve while any tier is untried, so a slow primary cannot spend the fallback's turn. See references/routing-and-budgets.md for a worked sizing example.

1. Break the circuit per provider. After a few consecutive failures, mark the provider open and skip it until a cooldown passes; any success closes it again. This stops the chain from spending its budget hammering a provider that is already down.

1. Validate output before anything downstream touches it. Every tier must return the same output type (structured output stays structured on every rung). Run the same schema check on each tier's answer and treat a validation failure as a transient error that moves to the next tier.

1. Write per-tier prompt variants. Do not run a prompt tuned for your strong model unchanged on a budget fallback: shorten the system prompt, tighten the output contract, drop tool-use expectations the weaker tier cannot meet. Carry which tier served the response in the result so the app can pick the right prompt, schema check, and eval. If a model was never validated against your prompt, do not silently substitute it. See references/degraded-mode-prompts.md.

1. Plan the honest degraded modes. Decide in advance what happens when every tier is exhausted: a cached or previously generated answer, a templated partial answer, queueing the request with an honest wait estimate, or escalating to a human. Write the user-facing message for each: what happened, what happens next, what the user can do now. Provider names and circuit state stay out of the UI; the reason goes to telemetry. See references/degraded-mode-prompts.md.

1. Log and alert like the chain matters. One log line per call (provider, tier, latency, tokens), one extra line per fallover, and an alert on fallback-trigger rate as its own signal. A fallback that fires constantly means the primary is degraded even when every request technically succeeds.

Output Contract

A fallback chain built with this skill ships:

• A route table: logical route names mapped to ordered tiers, recorded in config (numbers: attempts per tier, per-attempt timeout, whole-request deadline, fallback reserve, circuit threshold and cooldown).
• Per-tier prompt variants and one output validator shared by all tiers.
• A circuit breaker per provider with a stated cooldown.
• A degraded-mode plan: which fallback content each route uses and the exact user-facing message for each mode.
• Per-route logging (provider, tier, latency, tokens, fallback fired) plus a fallback-rate alert.
• A self-test against fault-injection stubs proving: retry then succeed, timeout fails over, open circuits are skipped, the reserve protects the fallback's turn, bad output fails over, fatal errors fail fast, and exhaustion degrades honestly.

Operating Rules

1. Chains stay short: primary plus one or two alternates. Every extra rung needs a written reason.
2. Fallback triggers on 429 explicitly, not only on 5xx and timeouts. A provider rate-limiting you is a provider failing you.
3. One deadline per request. No unbounded retries, ever.
4. Never retry non-idempotent steps. Never retry 4xx caller errors.
5. Every tier returns the same output type. A type mismatch at failover is a bug in the chain, not in the provider.
6. Never silently substitute an unvalidated model. A visible degraded message beats an invisible wrong answer.
7. Log every fallover. Alert on fallback rate separately from error rate.
8. Every factual claim about a provider (quotas, headers, SDK behavior) comes from that provider's docs or observed responses. Never invent numbers.

Tooling

bin/fallback_chain.py implements the pattern dependency-free and self-tests offline with fault-injection stubs:

python3 bin/fallback_chain.py --self-test

Import FallbackChain, Tier, CircuitBreaker, TransientError, FatalError, and ChainExhausted directly. Wrap your provider SDK calls in Tier.call functions. Set validate for output checks and on_exhausted for degraded answers.

References

• references/routing-and-budgets.md: choosing a routing rule, sizing attempts, timeouts, deadline, and reserve, with a worked example.
• references/degraded-mode-prompts.md: per-tier prompt variants, the honest-degradation ladder, user-facing message templates, and telemetry fields for degraded operation.