--- name: "api_polling_patterns" description: > Poll HTTP APIs efficiently: conditional requests with ETags and If-Modified-Since, cursor and since-timestamp pagination, dedup windows, backoff with jitter, and knowing when to stop polling and switch to webhooks. Trigger this when an agent needs to watch a REST endpoint for changes, sync items from an API on a timer, or walk a paginated feed without hammering the server. (HTTP client layer) metadata: includeInPrompt: true --- ## Purpose Polling is the HTTP client layer: one client, asking a server "anything new?" on a schedule, and making each ask as cheap and as safe as possible. This skill owns that discipline: conditional requests so unchanged data costs nothing, cursors so paging never skips or repeats, dedup so repeats never double-process, backoff so failures do not turn into a stampede, and a clear rule for when polling itself is the wrong tool. Not this skill: the cousins below sit at other layers and are NOT this skill. - message-queues: "Design async task queues: when to queue vs do work inline, producers/consumers, dead-letter queues, retry with exponential backoff + jitter, idempotent consumers, and at-least-once vs at-most-once delivery semantics. Triggers on 'should I queue this', 'add a job queue', 'dead letter queue', 'retry backoff', 'idempotent consumer', 'task queue design'." (server-side async infra layer) - n8n-workflow-automation: "Run n8n as agent infrastructure. Use when self-hosting n8n for production (Docker Compose, Postgres, queue mode), designing resilient workflows with retries and error workflows, or wiring credentials through external secrets vaults instead of hardcoding them." (orchestration platform layer) ## Workflow 1. Start with conditional requests. On each 200, store the `ETag` and `Last-Modified` values. On the next poll, send them back as `If-None-Match` and `If-Modified-Since`. A 304 means "nothing changed": the body is empty, so skip all processing for that poll. status, headers, body = get("/resource") etag = headers["ETag"] status, headers, body = get("/resource", {"If-None-Match": etag}) # status == 304, body empty: nothing to do Real fixture output, first fetch then three polls: 200, ETag="3afb8404574c5614f5b6495c04611b41" poll 1: 304 (0 bytes returned) poll 2: 304 (0 bytes returned) poll 3: 304 (0 bytes returned) 2. Page with cursors, never page numbers. Take `next_cursor` from each response and follow it until `has_more` is false. Cursors are opaque: never construct, decode, or guess them. If the API offers a since-timestamp filter instead, store the newest timestamp you have seen and pass it on the next poll. Real fixture output, full walk: 200 got ids [0, 1] next_cursor=c1 has_more=True 200 got ids [2, 3] next_cursor=c2 has_more=True 200 got ids [4, 5] next_cursor=None has_more=False 3. Dedup on your side. Keep a seen-set mapping each item id to the time you first saw it, and expire entries after a window (60s is a sane default). Mark ids as you consume each batch, not after, so duplicates inside a single response are caught too. Real fixture output on a duplicate-prone stream: poll 1: 200 saw [1, 2, 2] -> new [1, 2] poll 2: 200 saw [2, 2, 3] -> new [3] poll 3: 200 saw [2, 3, 4] -> new [4] The within-batch double of id 2 never reached the processor. 4. Back off with jitter on failures. For 429, 5xx, and network errors, sleep `min(2**n, cap) * uniform(0.75, 1.25)` seconds before retry n. Honor `Retry-After` when the server sends it. Never retry 4xx (except 429): a 400/401/403/404 means the request is wrong, so fix it instead. Real fixture output against a closed port: backoff: attempt 1, sleeping 2.22s (2**1 * jitter) backoff: attempt 2, sleeping 4.25s (2**2 * jitter) backoff: attempt 3, sleeping 9.77s (2**3 * jitter) recovered: 200 5. Decide whether polling is still the right call. Track two numbers: empty polls (304s plus polls with zero new items) and processed items. If empty polls reach your threshold, the change rate is too low to justify polling; subscribe to the API's webhooks instead. If processed items reach your threshold, the change volume is high enough that polling latency is costing you; subscribe instead. When you switch, reuse the same dedup window on deliveries, because webhooks arrive at-least-once. Real fixture output with thresholds of 5 empty polls: RECOMMEND WEBHOOKS: 5 empty polls (>= 5). Change rate is too low to justify polling; subscribe to events instead. ## Output Contract When this skill fires, the agent produces a polling plan or script that states all of the following, with concrete values: - interval and freshness target (e.g. "poll every 60s, changes seen within ~2 minutes") - which validators it stores (ETag, Last-Modified) and which it sends back - pagination method: cursor field names and termination condition, or the since-timestamp parameter name - dedup window length and the stable id field used as the dedup key - backoff base, cap, jitter range, and which statuses are retried vs fixed - webhook thresholds: the empty-poll count and processed-item count that trigger the switch recommendation Verification before claiming it works: a local run showing a 200 followed by 304s with empty bodies, a cursor walk that ends on `has_more: false`, and a duplicate-bearing batch where only new ids get processed. ## Operating Rules - A 304 has no body. Never parse, process, or store anything from it. - Store validators from every 200; send them back on every subsequent poll. - Cursors are opaque tokens. Never build or decode them; only follow them. - Dedup key is the item's stable id. Mark ids while consuming the batch. - Expire dedup entries so the set stays bounded; a 60s window is a sane default unless the API replays older data. - Retry only 429, 5xx, and network errors, with exponential backoff plus jitter and a cap. Fix 4xx instead of retrying them. - Respect `Retry-After` when present; it overrides your computed sleep. - Never shrink the interval as a fix. If you need fresher data than your thresholds allow, that is the webhook rule firing, not a reason to poll faster. - The webhook decision rule is mandatory: once either threshold is crossed, propose the switch to the user instead of continuing to poll. - Keep polling code to idempotent GETs only. Polling never creates, updates, or deletes server state. ## Reference poller (stdlib only) Save as `poller.py`, start the fixture below, then run `python3 poller.py`. Covers ETag caching, cursor pagination, the dedup window, backoff with jitter, and the switch-to-webhooks decision rule. Quoted outputs above come from this exact script. ```python """poller.py -- disciplined HTTP polling demo (stdlib only). Save as poller.py, then run: python3 poller.py Needs the fixture server first: python3 fixture.py (127.0.0.1:8901) """ import json, time, random, urllib.request, urllib.error BASE = "http://127.0.0.1:8901" SEEN = {} # id -> first-seen timestamp (dedup window) DEDUP_WINDOW = 60 # seconds an id stays "seen" MAX_BACKOFF = 30 # seconds cap WEBHOOK_AFTER_EMPTY = 5 # empty polls -> recommend webhooks WEBHOOK_AFTER_CHURNS = 8 # processed items above this -> recommend webhooks def get(path, headers=None): url = path if path.startswith("http") else BASE + path req = urllib.request.Request(url, headers=headers or {}) try: with urllib.request.urlopen(req, timeout=5) as r: return r.status, r.headers, r.read() except urllib.error.HTTPError as e: return e.code, e.headers, e.read() except Exception as e: return None, None, str(e).encode() def backoff(n): delay = min(2 ** n, MAX_BACKOFF) * random.uniform(0.75, 1.25) print(f" backoff: attempt {n}, sleeping {delay:.2f}s (2**{n} * jitter)") time.sleep(delay) def new_items(items): now = time.monotonic() for k in [k for k, t in SEEN.items() if now - t > DEDUP_WINDOW]: del SEEN[k] fresh = [] # mark-as-you-go: catches dupes within one batch too for it in items: if it["id"] not in SEEN: SEEN[it["id"]] = now fresh.append(it) return fresh def main(): print("== conditional GET ==") status, hdrs, body = get("/resource") etag = hdrs.get("ETag") print(f" first fetch: {status}, ETag={etag}, body={body.decode()}") empty = 0 for i in range(3): status, hdrs, body = get("/resource", {"If-None-Match": etag}) print(f" poll {i+1}: {status} ({len(body)} bytes returned)") empty += 1 if status == 304 else 0 print("== cursor walk ==") cursor, got = None, [] while True: p = "/items" + (f"?cursor={cursor}" if cursor else "") status, _, body = get(p) page = json.loads(body) got += page["items"] print(f" {status} got ids {[i['id'] for i in page['items']]}" f" next_cursor={page['next_cursor']} has_more={page['has_more']}") cursor = page["next_cursor"] if not page["has_more"]: break print(f" walked {len(got)} items across 3 pages") print("== stream with dedup window ==") processed = 0 for i in range(6): status, _, body = get("/stream") items = json.loads(body)["items"] fresh = new_items(items) print(f" poll {i+1}: {status} saw {[x['id'] for x in items]}" f" -> new {[x['id'] for x in fresh]}") if not fresh: empty += 1 processed += len(fresh) print("== backoff demo (closed port) ==") for n in range(1, 4): status, _, _ = get("http://127.0.0.1:9999/nope") if status is None: backoff(n) status, _, _ = get("/resource") print(f" recovered: {status}") print("== webhook decision ==") if empty >= WEBHOOK_AFTER_EMPTY: print(f" RECOMMEND WEBHOOKS: {empty} empty polls " f"(>= {WEBHOOK_AFTER_EMPTY}). Change rate is too low to justify " f"polling; subscribe to events instead.") elif processed >= WEBHOOK_AFTER_CHURNS: print(f" RECOMMEND WEBHOOKS: {processed} items processed " f"(>= {WEBHOOK_AFTER_CHURNS}). Change volume is high enough " f"that polling latency is costing you; subscribe instead.") else: print(f" keep polling: {empty} empty polls, {processed} items " f"processed. Neither threshold crossed.") if __name__ == "__main__": main() ``` ## Fixture server (local test harness) Local test harness ONLY, not production code. Save as `fixture.py`, run `python3 fixture.py`, and it listens on 127.0.0.1:8901. Serves the three endpoints the poller above was verified against. ```python """Local test harness ONLY (not production code). Run: python3 fixture.py (listens on 127.0.0.1:8901) Endpoints: GET /resource JSON doc with ETag + Last-Modified; honors If-None-Match / If-Modified-Since -> 304. GET /items?cursor= cursor-paginated: 3 pages, 2 items each. GET /stream occasionally repeats items (dedup test). """ import json, hashlib from http.server import BaseHTTPRequestHandler, HTTPServer PORT = 8901 BODY = json.dumps({"widget": "sprocket", "rev": 7, "status": "stable"}).encode() ETAG = '"%s"' % hashlib.md5(BODY).hexdigest() LMOD = "Wed, 07 Oct 2026 12:00:00 GMT" ITEMS = [{"id": i, "v": f"row-{i}"} for i in range(6)] STREAM = [1, 2, 2, 3, 4, 4, 5] class H(BaseHTTPRequestHandler): def log_message(self, *a): pass def _json(self, code, obj, extra=()): raw = json.dumps(obj).encode() self.send_response(code) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(raw))) for k, v in extra: self.send_header(k, v) self.end_headers() if code != 304: self.wfile.write(raw) def do_GET(self): path = self.path.split("?", 1)[0] qs = self.path.split("?", 1)[1] if "?" in self.path else "" if path == "/resource": inm = self.headers.get("If-None-Match") ims = self.headers.get("If-Modified-Since") if inm == ETAG or ims == LMOD: self.send_response(304) self.send_header("ETag", ETAG) self.send_header("Last-Modified", LMOD) self.end_headers() return self._json(200, json.loads(BODY), [("ETag", ETAG), ("Last-Modified", LMOD)]) elif path == "/items": cur = "" for kv in qs.split("&"): if kv.startswith("cursor="): cur = kv.split("=", 1)[1] idx = {"": 0, "c1": 2, "c2": 4}.get(cur) if idx is None: self._json(400, {"error": "bad cursor"}); return page = ITEMS[idx:idx + 2] nxt = {0: "c1", 2: "c2"}.get(idx) self._json(200, {"items": page, "next_cursor": nxt, "has_more": nxt is not None}) elif path == "/stream": n = self.server.seen self.server.seen += 1 win = STREAM[n % len(STREAM): n % len(STREAM) + 3] self._json(200, {"items": [{"id": i, "v": f"s{i}"} for i in win]}) else: self._json(404, {"error": "nope"}) srv = HTTPServer(("127.0.0.1", PORT), H) srv.seen = 0 print("fixture up on 127.0.0.1:%d" % PORT, flush=True) srv.serve_forever() ```