firecrawl_scraping_patterns v1.0.0

name: "firecrawl_scraping_patterns" description: "Run web scraping with the Firecrawl API the disciplined way: pick scrape vs map vs crawl vs extract by job shape, size every crawl with map-first, scope crawls with path regexes and depth limits, write tight extraction schemas, and keep rate and legal hygiene. Triggers when an agent needs to pull data off the public web through Firecrawl."

Firecrawl Scraping Patterns

Purpose

Firecrawl turns pages into LLM-ready markdown or structured JSON and handles JS rendering, bot evasion, and PDF parsing so you do not have to. This skill covers the judgment part: which endpoint to use, how to scope a crawl so it does not burn your whole credit budget, how to extract structured data with schemas, and how to stay polite and legal. Details live in references/; this file is the workflow.

Workflow

1. Size before you spend: map first. Before any crawl, run map on the target. It is cheap (1 credit per page discovered, no content fetched) and returns link objects with URLs, titles, and descriptions in v2. A map tells you how big the site is and which sections matter, so your crawl limit and path filters are real numbers, not guesses.

1. Pick the endpoint by job shape.

| Job | Endpoint | Why | |---|---|---| | One known page | /v2/scrape (sync) | Cheapest single fetch | | A known list of URLs | /v2/batch/scrape (async) | One job, one poll loop | | Every page of a site or section | /v2/crawl (async) | Recursive, but map first | | Same fields across many pages | /v2/extract (async) | Prompt plus JSON schema, glob URLs | | No known URLs, need an answer | /v2/search | Web search, optionally scrapes results inline | | Just the URL list | /v2/map | Discovery only, fast |

Never crawl when map plus a batch scrape of the URLs you actually need will do. Crawls are the most expensive thing here because every page costs a credit.

1. Scope the crawl. Start from the map output and set, in this order:

• includePaths to keep only the section you want (Rust RE2 regex on the pathname; note the start URL must match too, or you get zero pages). excludePaths for traps like calendars, search result pages, and tag archives.
• limit sized from the map count, never the 10,000 default.
• maxDiscoveryDepth small (1 to 3 for a section, more only with a tight includePaths).
• sitemap: "include" for coverage, "skip" if the sitemap is stale, "only" for a quick pass over known pages.
• deduplicateSimilarURLs: true, ignoreQueryParameters: true unless query params carry real content.
• scrapeOptions carries the scrape settings for every page: formats: ["markdown"] and onlyMainContent: true by default; add links when you need the link graph, json when you want structured output per page.

1. Poll async jobs correctly. /v2/crawl, /v2/batch/scrape, and /v2/extract return {success, id} (extract also returns invalidURLs). Poll GET /v2/crawl/{id} until status is completed or failed (terminal states; scraping means still running). Status responses are paginated with a next cursor; walk it for large jobs. Check /v2/crawl/{id}/errors for per-URL failures and the EXTERNAL_LINK skip codes. DELETE /v2/crawl/{id} cancels a runaway job. For long jobs, set a webhook instead of polling.

1. Write extraction schemas like a contract. For one page, use formats: [{type: "json", schema, prompt}] on scrape. For many URLs, use /v2/extract with urls (glob format), a plain-language prompt, and a schema that conforms to JSON Schema. Keep schemas narrow: the fields you will actually consume. Root the schema at an object; inspect the returned shape before assuming it. With ignoreInvalidURLs: true (default), bad URLs land in invalidURLs instead of failing the job, so always read that field. Add showSources: true when you need to cite where each value came from.

1. Keep rate discipline. Rate limits are per team over a sliding 60-second window, so every key on the team shares one counter. Set delay (seconds between requests; this forces concurrency to 1) on unfamiliar hosts, or maxConcurrency for a gentler cap. A 429 means back off and retry with spacing; a 402 means the credit balance is empty. Never hammer the status endpoint on a fast loop: statuses poll at generous limits, but a webhook beats polling for jobs that run minutes. Watch the credit meter on the dashboard before large jobs, and remember you are charged once a page is fetched successfully even if the site itself answers with a 4xx or 5xx.

1. Keep legal hygiene. Firecrawl respects robots.txt by default; do not reach for ignoreRobotsTxt (Enterprise-only, and trying it on a lower plan is not a workaround, it is a violation). Check the target's robots.txt and terms yourself before scraping: some platforms ban scraping outright, and Firecrawl's own terms prohibit unlawful use and use that violates laws, regulations, or contractual provisions, with the liability landing on you. Scrape public pages only. Never use scraping to bypass login, paywalls-as-access-control, or anti-bot gates on sites that do not want you there. Keep the request rate polite, attribute what you quote, treat scraped personal data as subject to GDPR and similar regimes, and respect copyright: take facts and short quotes, not whole articles.

Output Contract

When you finish a scraping job, report: (a) the endpoint chosen and why, (b) the scope config used (paths, depth, limit), (c) estimated vs actual credit spend, (d) error summary (failed URLs, skip codes), (e) the data delivered, with sources where extract was used.

Operating Rules

• Map before crawl, always. A crawl without a preceding map needs a written reason.
• Never run a crawl with default limit. Every crawl gets an explicit limit sized from a map.
• Keep the Firecrawl key in FIRECRAWL_API_KEY, never in code, logs, or pasted output. Keys start with fc-.
• On 429, wait and retry with backoff. On 402, stop and report. On a crawl that is clearly wrong (wrong section, exploding page count), cancel it instead of letting it finish.
• Public pages only. No credentials, no auth bypass, no stealth tricks aimed at sites that block you. If a site needs stealth proxy to load at all, note that it costs 5 credits per request when actively used, and ask whether the data is worth it.
• Every factual claim about Firecrawl's API, pricing, or limits in your own notes must be traceable to docs.firecrawl.dev or a verified fetch, dated. Numbers change; do not repeat last quarter's table from memory.