---
name: "firecrawl_scraping_patterns"
description: "Run web scraping with the Firecrawl API the disciplined way: pick scrape vs map vs crawl vs extract by job shape, size every crawl with map-first, scope crawls with path regexes and depth limits, write tight extraction schemas, and keep rate and legal hygiene. Triggers when an agent needs to pull data off the public web through Firecrawl."
---

# Firecrawl Scraping Patterns

## Purpose
Firecrawl turns pages into LLM-ready markdown or structured JSON and handles JS rendering, bot evasion, and PDF parsing so you do not have to. This skill covers the judgment part: which endpoint to use, how to scope a crawl so it does not burn your whole credit budget, how to extract structured data with schemas, and how to stay polite and legal. Details live in `references/`; this file is the workflow.

## Workflow

1. **Size before you spend: map first.** Before any crawl, run `map` on the target. It is cheap (1 credit per page discovered, no content fetched) and returns link objects with URLs, titles, and descriptions in v2. A map tells you how big the site is and which sections matter, so your crawl `limit` and path filters are real numbers, not guesses.

2. **Pick the endpoint by job shape.**

   | Job | Endpoint | Why |
   |---|---|---|
   | One known page | `/v2/scrape` (sync) | Cheapest single fetch |
   | A known list of URLs | `/v2/batch/scrape` (async) | One job, one poll loop |
   | Every page of a site or section | `/v2/crawl` (async) | Recursive, but map first |
   | Same fields across many pages | `/v2/extract` (async) | Prompt plus JSON schema, glob URLs |
   | No known URLs, need an answer | `/v2/search` | Web search, optionally scrapes results inline |
   | Just the URL list | `/v2/map` | Discovery only, fast |

   Never crawl when map plus a batch scrape of the URLs you actually need will do. Crawls are the most expensive thing here because every page costs a credit.

3. **Scope the crawl.** Start from the map output and set, in this order:
   - `includePaths` to keep only the section you want (Rust RE2 regex on the pathname; note the start URL must match too, or you get zero pages). `excludePaths` for traps like calendars, search result pages, and tag archives.
   - `limit` sized from the map count, never the 10,000 default.
   - `maxDiscoveryDepth` small (1 to 3 for a section, more only with a tight includePaths).
   - `sitemap: "include"` for coverage, `"skip"` if the sitemap is stale, `"only"` for a quick pass over known pages.
   - `deduplicateSimilarURLs: true`, `ignoreQueryParameters: true` unless query params carry real content.
   - `scrapeOptions` carries the scrape settings for every page: `formats: ["markdown"]` and `onlyMainContent: true` by default; add `links` when you need the link graph, `json` when you want structured output per page.

4. **Poll async jobs correctly.** `/v2/crawl`, `/v2/batch/scrape`, and `/v2/extract` return `{success, id}` (extract also returns `invalidURLs`). Poll `GET /v2/crawl/{id}` until status is `completed` or `failed` (terminal states; `scraping` means still running). Status responses are paginated with a `next` cursor; walk it for large jobs. Check `/v2/crawl/{id}/errors` for per-URL failures and the `EXTERNAL_LINK` skip codes. `DELETE /v2/crawl/{id}` cancels a runaway job. For long jobs, set a `webhook` instead of polling.

5. **Write extraction schemas like a contract.** For one page, use `formats: [{type: "json", schema, prompt}]` on scrape. For many URLs, use `/v2/extract` with `urls` (glob format), a plain-language `prompt`, and a `schema` that conforms to JSON Schema. Keep schemas narrow: the fields you will actually consume. Root the schema at an object; inspect the returned shape before assuming it. With `ignoreInvalidURLs: true` (default), bad URLs land in `invalidURLs` instead of failing the job, so always read that field. Add `showSources: true` when you need to cite where each value came from.

6. **Keep rate discipline.** Rate limits are per team over a sliding 60-second window, so every key on the team shares one counter. Set `delay` (seconds between requests; this forces concurrency to 1) on unfamiliar hosts, or `maxConcurrency` for a gentler cap. A `429` means back off and retry with spacing; a `402` means the credit balance is empty. Never hammer the status endpoint on a fast loop: statuses poll at generous limits, but a webhook beats polling for jobs that run minutes. Watch the credit meter on the dashboard before large jobs, and remember you are charged once a page is fetched successfully even if the site itself answers with a 4xx or 5xx.

7. **Keep legal hygiene.** Firecrawl respects `robots.txt` by default; do not reach for `ignoreRobotsTxt` (Enterprise-only, and trying it on a lower plan is not a workaround, it is a violation). Check the target's `robots.txt` and terms yourself before scraping: some platforms ban scraping outright, and Firecrawl's own terms prohibit unlawful use and use that violates laws, regulations, or contractual provisions, with the liability landing on you. Scrape public pages only. Never use scraping to bypass login, paywalls-as-access-control, or anti-bot gates on sites that do not want you there. Keep the request rate polite, attribute what you quote, treat scraped personal data as subject to GDPR and similar regimes, and respect copyright: take facts and short quotes, not whole articles.

## Output Contract
When you finish a scraping job, report: (a) the endpoint chosen and why, (b) the scope config used (paths, depth, limit), (c) estimated vs actual credit spend, (d) error summary (failed URLs, skip codes), (e) the data delivered, with sources where extract was used.

## Operating Rules
- Map before crawl, always. A crawl without a preceding map needs a written reason.
- Never run a crawl with default `limit`. Every crawl gets an explicit limit sized from a map.
- Keep the Firecrawl key in `FIRECRAWL_API_KEY`, never in code, logs, or pasted output. Keys start with `fc-`.
- On `429`, wait and retry with backoff. On `402`, stop and report. On a crawl that is clearly wrong (wrong section, exploding page count), cancel it instead of letting it finish.
- Public pages only. No credentials, no auth bypass, no stealth tricks aimed at sites that block you. If a site needs stealth proxy to load at all, note that it costs 5 credits per request when actively used, and ask whether the data is worth it.
- Every factual claim about Firecrawl's API, pricing, or limits in your own notes must be traceable to docs.firecrawl.dev or a verified fetch, dated. Numbers change; do not repeat last quarter's table from memory.