scrape_resilient v1.0.0

name: "scrape_resilient" description: "Scrape the web reliably: retries with backoff, defensive parsing, bot-protection handling, caching, and politeness."

Resilient Scraping

Getting a page once is easy. Getting it a thousand times — across site redesigns, rate limits, and bot protection — without getting blocked or serving stale data, is the actual job.

Workflow

1. Fetch defensively. The network is hostile; code like it:

• Retry with exponential backoff + jitter: 1s, 2s, 4s… cap at ~60s, max 4–5 attempts. Jitter prevents thundering herds.
• Retry on 429, 502, 503, 504 and on connection errors. Do NOT retry on 400, 401, 403, 404 — those are answers, not outages. (Exception: some CDNs 403 bot-like clients; see step 3.)
• Set timeouts on everything. A fetch with no timeout is a hang. Defaults: connect 10s, read 30s.
• Send a real User-Agent. Some endpoints reject library default UAs outright while curl or a browser UA passes fine:

python
     headers = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"}
     

• If a site behaves differently between HTTP clients, the difference is almost always headers or TLS fingerprinting — try curl or requests with a browser UA before concluding the endpoint is down.

1. Parse for the structure, not the snapshot.

• Select by semantic hooks (article, [data-*] attributes, headings), not div.div.div.span:nth-child(4) chains that die on the next deploy.
• Prefer structured sources first: JSON endpoints, RSS/Atom feeds, sitemaps, application/ld+json blobs. HTML scraping is the fallback.
• Validate what you extracted: expected fields present, types sane, counts within reason. A parser that returns None silently is a lie waiting to happen — fail loudly on unexpected shapes.
• Pin a sample of raw HTML per source. When parsing breaks, diff against the pinned sample to see what the site changed.

1. Handle bot protection honestly.

• Read robots.txt and honor crawl-delay. It's the site's stated terms; ignoring it is how you get IP-banned.
• A 403 is a verdict until proven otherwise. Change the client (headers/UA) once; if it still 403s, the site doesn't want you there — stop.
• If you hit a CAPTCHA or hard block: stop and surface it. Do not rotate IPs to evade a block — that's the line between scraping and attacking.
• Prefer official APIs when they exist, even paid ones, for anything you depend on daily. Scraping is a bridge, not a foundation.

1. Cache aggressively, revalidate cheaply.

• Cache responses on disk keyed by URL + params. Serve from cache inside retry loops so a transient failure doesn't cost you data you already had.
• Use ETag / If-Modified-Since when the server supports it — a 304 is cheaper than a full page.
• Never cache error responses as if they were data. Cache 200s; retry 5xxs.

1. Be polite by default.

• One request at a time per host unless you know its limits. Space bursts with small sleeps.
• Identify yourself in the UA string if you're crawling at volume.
• Scrape the minimum you need. Fetching 10,000 pages to use 50 rows is waste with a side of ban risk.

Output Contract

Scraped data ships with: source URL, fetch timestamp, the parser version or selector set used, and a validation note (what was checked). Stale or partial data is labeled as such — never silently.

Operating Rules

1. Never trust a single fetch for data someone will act on. Re-fetch or cross-check against a second source.
2. Log every fetch: URL, status, duration, bytes. When something breaks at 3am, this log is the diagnosis.
3. Timeouts on every request, no exceptions.
4. If the data matters daily, the scraper gets a monitor: alert when field counts drop or the shape changes, not when a human notices.
5. Treat a shape change in the HTML as a breaking change in your parser — because it is.