api_rate_limit_design v1.0.0

name: api_rate_limit_design description: Designs client-side rate-limit discipline for APIs: self-throttling token buckets, exponential backoff with jitter, 429 handling, per-key quotas, and load-testing retry logic before a provider throttles you mid-run. Use when writing an API client, SDK, or worker that must survive strict provider rate limits.

API Rate Limit Design

Purpose

Make API clients behave well under rate limits: never fire a request you expect to be rejected, and when rejection happens, back off in a way that does not make the problem worse. This skill covers the client side only (self-throttle, backoff, retry policy), not server-side enforcement.

Workflow

1. Find the limit. Read the provider's docs for the exact quota (requests/sec, requests/min, per-key vs per-host). If no docs exist, start conservative and read X-RateLimit- / RateLimit- headers opportunistically. Do not assume any API emits them.

1. Self-throttle with a token bucket. Run a local TokenBucket at 80-90% of the provider's stated rate, with capacity sized to the burst you actually need. One bucket per quota key (host, API key, or endpoint) via BucketRegistry. The burst of the first 5 permits at 10 req/s with capacity 5 flows instantly; the 6th waits ~100ms. Pacing below the limit is cheaper than eating 429s.

1. Retry only what is retryable. Retry 429 and 500/502/503/504 plus transient network errors. Fail immediately on 400, 401, 403, 404, 422. Never retry a request that failed validation.

1. Honor Retry-After first. If the response carries Retry-After, wait that long (parse both forms: delta-seconds and HTTP-date; clamp to >= 0). A server-supplied wait always beats a computed one, and if both RateLimit and Retry-After fields appear, Retry-After takes precedence.

1. Back off with full jitter otherwise. Default formula: sleep = random(0, min(cap, base * 2^attempt)). Exponential backoff alone synchronizes retrying clients into lockstep waves; jitter is what actually spreads the load.

1. Cap the total damage. Set a retry budget (total seconds allowed to spend waiting across attempts). When it empties, raise and fail fast rather than feeding a retry storm. Retry at exactly one layer of the stack: layers multiply, so 3 retries at 3 layers is up to 27 attempts.

1. Load-test the retry logic before shipping. Hammer the client against a stub that returns 429s with and without Retry-After, then check the request timeline: no synchronized retry waves, no instant retries on an HTTP-date Retry-After, and the budget trips instead of hanging.

Output Contract

Any client built with this skill ships:

• A pacing layer (token bucket, per quota key) with rate and capacity recorded in config, not hardcoded.
• A retry loop: retryable-status list, Retry-After precedence, full-jitter backoff with stated base/cap, max attempts, and a total retry budget.
• A Retry-After parser handling both RFC 9110 forms, clamped >= 0.
• A self-test or load-test proving: 429s are honored, jitter stays in bounds, non-retryable statuses fail fast, and budget exhaustion raises instead of sleeping forever.

Operating Rules

1. Never retry non-retryable statuses (4xx client errors), period.
2. Never assume a rate-limit header family exists; read opportunistically, self-throttle by default.
3. Server pushback (Retry-After) always wins over computed backoff.
4. Full jitter is the default backoff. Document it when you pick equal or decorrelated jitter instead.
5. No retry policy ships without a budget and a one-layer rule. "Added backoff" without those is an availability risk, not resilience.
6. Every factual claim about a provider's limits comes from that provider's docs or observed headers. Never invent numbers.

Tooling

bin/rate_limit.py implements the whole pattern dependency-free and self-tests offline:

python3 bin/rate_limit.py --self-test

Import TokenBucket, BucketRegistry, parse_retry_after, full_jitter, and call_with_retry directly when building a client.

References

• references/backoff-formulas.md: the three jitter strategies, Brooker's verdict, parameter guidance.
• references/retry-after-and-headers.md: 429, Retry-After forms, header families, caching rule.
• references/testing-retry-logic.md: how to load-test the retry loop against a stub.