---
name: "scrape_resilient"
description: "Scrape the web reliably: retries with backoff, defensive parsing, bot-protection handling, caching, and politeness."
---

# Resilient Scraping

Getting a page once is easy. Getting it a thousand times — across site
redesigns, rate limits, and bot protection — without getting blocked or
serving stale data, is the actual job.

## Workflow

1. **Fetch defensively.** The network is hostile; code like it:
   - Retry with exponential backoff + jitter: 1s, 2s, 4s… cap at ~60s,
     max 4–5 attempts. Jitter prevents thundering herds.
   - Retry on 429, 502, 503, 504 and on connection errors. Do NOT retry
     on 400, 401, 403, 404 — those are answers, not outages. (Exception:
     some CDNs 403 bot-like clients; see step 3.)
   - Set timeouts on everything. A fetch with no timeout is a hang.
     Defaults: connect 10s, read 30s.
   - Send a real `User-Agent`. Some endpoints reject library default UAs
     outright while curl or a browser UA passes fine:
     ```python
     headers = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"}
     ```
   - If a site behaves differently between HTTP clients, the difference
     is almost always headers or TLS fingerprinting — try curl or
     `requests` with a browser UA before concluding the endpoint is down.

2. **Parse for the structure, not the snapshot.**
   - Select by semantic hooks (`article`, `[data-*]` attributes,
     headings), not `div.div.div.span:nth-child(4)` chains that die on
     the next deploy.
   - Prefer structured sources first: JSON endpoints, RSS/Atom feeds,
     sitemaps, `application/ld+json` blobs. HTML scraping is the fallback.
   - Validate what you extracted: expected fields present, types sane,
     counts within reason. A parser that returns `None` silently is a lie
     waiting to happen — fail loudly on unexpected shapes.
   - Pin a sample of raw HTML per source. When parsing breaks, diff
     against the pinned sample to see what the site changed.

3. **Handle bot protection honestly.**
   - Read `robots.txt` and honor crawl-delay. It's the site's stated
     terms; ignoring it is how you get IP-banned.
   - A 403 is a verdict until proven otherwise. Change the client
     (headers/UA) once; if it still 403s, the site doesn't want you
     there — stop.
   - If you hit a CAPTCHA or hard block: stop and surface it. Do not
     rotate IPs to evade a block — that's the line between scraping and
     attacking.
   - Prefer official APIs when they exist, even paid ones, for anything
     you depend on daily. Scraping is a bridge, not a foundation.

4. **Cache aggressively, revalidate cheaply.**
   - Cache responses on disk keyed by URL + params. Serve from cache
     inside retry loops so a transient failure doesn't cost you data you
     already had.
   - Use `ETag` / `If-Modified-Since` when the server supports it — a
     304 is cheaper than a full page.
   - Never cache error responses as if they were data. Cache 200s;
     retry 5xxs.

5. **Be polite by default.**
   - One request at a time per host unless you know its limits. Space
     bursts with small sleeps.
   - Identify yourself in the UA string if you're crawling at volume.
   - Scrape the minimum you need. Fetching 10,000 pages to use 50 rows
     is waste with a side of ban risk.

## Output Contract

Scraped data ships with: source URL, fetch timestamp, the parser version
or selector set used, and a validation note (what was checked). Stale or
partial data is labeled as such — never silently.

## Operating Rules

1. Never trust a single fetch for data someone will act on. Re-fetch or
   cross-check against a second source.
2. Log every fetch: URL, status, duration, bytes. When something breaks
   at 3am, this log is the diagnosis.
3. Timeouts on every request, no exceptions.
4. If the data matters daily, the scraper gets a monitor: alert when
   field counts drop or the shape changes, not when a human notices.
5. Treat a shape change in the HTML as a breaking change in your parser —
   because it is.