---
name: llm_fallback_chains
description: Designs resilient LLM model routing: tiered fallback chains, routing rules, per-request retry budgets, circuit breakers, and degraded-mode prompts. Use when an app calls one or more LLM providers and needs to survive rate limits, timeouts, and outages without hanging, silently degrading, or double-firing side effects.
---

# LLM Fallback Chains

## Purpose

Keep an LLM-powered feature working when providers misbehave. One
provider down, rate-limiting, or timing out should move traffic to the
next tier in milliseconds and tell you it happened, never hang the user
and never pretend a weaker answer is the normal one.

This skill covers routing (which model serves which request), the chain
itself (retries, timeouts, one shared deadline), circuit breaking, and
what the product does when every model is unavailable. For client-side
rate-limit pacing alone, see `api_rate_limit_design`.

## Workflow

1. **Name logical routes, not provider strings.** Route code through
   logical tier names like `smart`, `fast`, `budget`. Reordering or
   swapping providers then becomes a config change, never a code change.

2. **Pick a routing rule.** Simplest first:
   - Rule-based: classify the task (extraction, chat, reasoning) and
     assign it to a tier. Cheapest option, zero added latency.
   - Complexity heuristic: short structured request goes to the budget
     tier; long open-ended request goes to the strong tier.
   - Cascade: try the budget tier first; escalate only when confidence
     is low or the output fails validation.
   - Managed: use a gateway (LiteLLM proxy, OpenRouter, or similar) for
     routing, fallback, and observability out of the box when you want
     less custom code.
   See `references/routing-and-budgets.md` for how to choose.

3. **Build the chain short and ordered.** Primary, then one or two
   known-good alternates, then a hard stop. Long chains add latency
   without adding reliability and hide which provider actually served
   the request. A different provider at the same tier beats a weaker
   model at the same provider when the problem is an outage.

4. **Retry the tier, then fail over.** Within one tier: retry transient
   failures (429, 5xx, timeouts, network blips) with exponential backoff
   and jitter, bounded attempts. Then move to the next tier. Never retry
   4xx caller errors or auth failures. Never retry a step whose first
   attempt already acted (sent email, wrote a record, charged a card):
   retry the model call, not the side effects.

5. **Set one deadline per request, not N timeouts.** Give each request a
   single wall-clock budget. Every attempt's timeout is the smaller of
   its per-attempt timeout and the remaining budget, and an attempt that
   cannot fit in what is left never starts. Withhold a fallback reserve
   while any tier is untried, so a slow primary cannot spend the
   fallback's turn. See `references/routing-and-budgets.md` for a
   worked sizing example.

6. **Break the circuit per provider.** After a few consecutive failures,
   mark the provider open and skip it until a cooldown passes; any
   success closes it again. This stops the chain from spending its
   budget hammering a provider that is already down.

7. **Validate output before anything downstream touches it.** Every tier
   must return the same output type (structured output stays structured
   on every rung). Run the same schema check on each tier's answer and
   treat a validation failure as a transient error that moves to the
   next tier.

8. **Write per-tier prompt variants.** Do not run a prompt tuned for your
   strong model unchanged on a budget fallback: shorten the system
   prompt, tighten the output contract, drop tool-use expectations the
   weaker tier cannot meet. Carry which tier served the response in the
   result so the app can pick the right prompt, schema check, and eval.
   If a model was never validated against your prompt, do not silently
   substitute it. See `references/degraded-mode-prompts.md`.

9. **Plan the honest degraded modes.** Decide in advance what happens
   when every tier is exhausted: a cached or previously generated
   answer, a templated partial answer, queueing the request with an
   honest wait estimate, or escalating to a human. Write the user-facing
   message for each: what happened, what happens next, what the user
   can do now. Provider names and circuit state stay out of the UI;
   the reason goes to telemetry. See `references/degraded-mode-prompts.md`.

10. **Log and alert like the chain matters.** One log line per call
    (provider, tier, latency, tokens), one extra line per fallover, and
    an alert on fallback-trigger rate as its own signal. A fallback
    that fires constantly means the primary is degraded even when every
    request technically succeeds.

## Output Contract

A fallback chain built with this skill ships:

- A route table: logical route names mapped to ordered tiers, recorded
  in config (numbers: attempts per tier, per-attempt timeout,
  whole-request deadline, fallback reserve, circuit threshold and
  cooldown).
- Per-tier prompt variants and one output validator shared by all tiers.
- A circuit breaker per provider with a stated cooldown.
- A degraded-mode plan: which fallback content each route uses and the
  exact user-facing message for each mode.
- Per-route logging (provider, tier, latency, tokens, fallback fired)
  plus a fallback-rate alert.
- A self-test against fault-injection stubs proving: retry then
  succeed, timeout fails over, open circuits are skipped, the reserve
  protects the fallback's turn, bad output fails over, fatal errors
  fail fast, and exhaustion degrades honestly.

## Operating Rules

1. Chains stay short: primary plus one or two alternates. Every extra
   rung needs a written reason.
2. Fallback triggers on 429 explicitly, not only on 5xx and timeouts.
   A provider rate-limiting you is a provider failing you.
3. One deadline per request. No unbounded retries, ever.
4. Never retry non-idempotent steps. Never retry 4xx caller errors.
5. Every tier returns the same output type. A type mismatch at failover
   is a bug in the chain, not in the provider.
6. Never silently substitute an unvalidated model. A visible degraded
   message beats an invisible wrong answer.
7. Log every fallover. Alert on fallback rate separately from error rate.
8. Every factual claim about a provider (quotas, headers, SDK behavior)
   comes from that provider's docs or observed responses. Never invent
   numbers.

## Tooling

`bin/fallback_chain.py` implements the pattern dependency-free and
self-tests offline with fault-injection stubs:

```
python3 bin/fallback_chain.py --self-test
```

Import `FallbackChain`, `Tier`, `CircuitBreaker`, `TransientError`,
`FatalError`, and `ChainExhausted` directly. Wrap your provider SDK
calls in `Tier.call` functions. Set `validate` for output checks and
`on_exhausted` for degraded answers.

## References

- `references/routing-and-budgets.md`: choosing a routing rule, sizing
  attempts, timeouts, deadline, and reserve, with a worked example.
- `references/degraded-mode-prompts.md`: per-tier prompt variants, the
  honest-degradation ladder, user-facing message templates, and
  telemetry fields for degraded operation.