---
name: api_rate_limit_design
description: Designs client-side rate-limit discipline for APIs: self-throttling token buckets, exponential backoff with jitter, 429 handling, per-key quotas, and load-testing retry logic before a provider throttles you mid-run. Use when writing an API client, SDK, or worker that must survive strict provider rate limits.
---

# API Rate Limit Design

## Purpose

Make API clients behave well under rate limits: never fire a request you
expect to be rejected, and when rejection happens, back off in a way that
does not make the problem worse. This skill covers the client side only
(self-throttle, backoff, retry policy), not server-side enforcement.

## Workflow

1. **Find the limit.** Read the provider's docs for the exact quota
   (requests/sec, requests/min, per-key vs per-host). If no docs exist,
   start conservative and read `X-RateLimit-*` / `RateLimit-*` headers
   opportunistically. Do not assume any API emits them.

2. **Self-throttle with a token bucket.** Run a local `TokenBucket` at
   80-90% of the provider's stated rate, with capacity sized to the burst
   you actually need. One bucket per quota key (host, API key, or endpoint)
   via `BucketRegistry`. The burst of the first 5 permits at 10 req/s with
   capacity 5 flows instantly; the 6th waits ~100ms. Pacing below the limit
   is cheaper than eating 429s.

3. **Retry only what is retryable.** Retry 429 and 500/502/503/504 plus
   transient network errors. Fail immediately on 400, 401, 403, 404, 422.
   Never retry a request that failed validation.

4. **Honor Retry-After first.** If the response carries `Retry-After`, wait
   that long (parse both forms: delta-seconds and HTTP-date; clamp to >= 0).
   A server-supplied wait always beats a computed one, and if both
   `RateLimit` and `Retry-After` fields appear, `Retry-After` takes
   precedence.

5. **Back off with full jitter otherwise.** Default formula:
   `sleep = random(0, min(cap, base * 2^attempt))`. Exponential backoff
   alone synchronizes retrying clients into lockstep waves; jitter is what
   actually spreads the load.

6. **Cap the total damage.** Set a retry budget (total seconds allowed to
   spend waiting across attempts). When it empties, raise and fail fast
   rather than feeding a retry storm. Retry at exactly one layer of the
   stack: layers multiply, so 3 retries at 3 layers is up to 27 attempts.

7. **Load-test the retry logic before shipping.** Hammer the client against
   a stub that returns 429s with and without `Retry-After`, then check the
   request timeline: no synchronized retry waves, no instant retries on an
   HTTP-date `Retry-After`, and the budget trips instead of hanging.

## Output Contract

Any client built with this skill ships:

- A pacing layer (token bucket, per quota key) with rate and capacity
  recorded in config, not hardcoded.
- A retry loop: retryable-status list, `Retry-After` precedence, full-jitter
  backoff with stated base/cap, max attempts, and a total retry budget.
- A `Retry-After` parser handling both RFC 9110 forms, clamped >= 0.
- A self-test or load-test proving: 429s are honored, jitter stays in
  bounds, non-retryable statuses fail fast, and budget exhaustion raises
  instead of sleeping forever.

## Operating Rules

1. Never retry non-retryable statuses (4xx client errors), period.
2. Never assume a rate-limit header family exists; read opportunistically,
   self-throttle by default.
3. Server pushback (`Retry-After`) always wins over computed backoff.
4. Full jitter is the default backoff. Document it when you pick equal or
   decorrelated jitter instead.
5. No retry policy ships without a budget and a one-layer rule. "Added
   backoff" without those is an availability risk, not resilience.
6. Every factual claim about a provider's limits comes from that provider's
   docs or observed headers. Never invent numbers.

## Tooling

`bin/rate_limit.py` implements the whole pattern dependency-free and
self-tests offline:

```
python3 bin/rate_limit.py --self-test
```

Import `TokenBucket`, `BucketRegistry`, `parse_retry_after`,
`full_jitter`, and `call_with_retry` directly when building a client.

## References

- `references/backoff-formulas.md`: the three jitter strategies, Brooker's
  verdict, parameter guidance.
- `references/retry-after-and-headers.md`: 429, `Retry-After` forms, header
  families, caching rule.
- `references/testing-retry-logic.md`: how to load-test the retry loop
  against a stub.