--- name: web_source_archiving description: Archive web sources so citations still resolve later: capture each cited page in at least two places (the Wayback Machine, Archive.today, and a local ArchiveBox copy where the claim matters), run a capture completeness check for the exact phrases and images the citation rests on, and file a sidecar record with the original URL, capture URLs, capture dates, and a SHA-256 hash. Use when someone cites a web page in research, journalism, a report, a paper, or a legal or policy draft, when a cited link has rotted or the page has changed, or when a bibliography needs archived-at links. For checking whether a citation supports a claim use citation integrity auditing; for general source evaluation use research methods; this skill is the capture and proof step. --- # Web Source Archiving ## Purpose A citation to a web page fails in two ways. The link rots: the URL returns an error, the site is restructured, or the page is deleted. Or the page stays up and changes, so the words the citation rests on are no longer there, while a plain link checker still reports the URL as fine. Both failures strand a reader who tries to verify the claim. This skill prevents both by treating capture as part of citing, not as cleanup after publication. Every web source that carries a claim gets captured at the moment it is cited, in at least two independent places, then checked for the exact text and images the claim needs, then recorded in a sidecar file that travels with the draft. A capture that was never checked is a hope, not a record. Services, endpoints, and failure shapes are in `references/service-notes.md`. A full rescue of a changed page is worked in `references/rescue-example.md`. ## Workflow 1. **Name the claim before capturing anything.** For each web source, write one sentence: the claim this source supports, plus the exact phrases on the page the claim rests on, quoted word for word. Usually one to three phrases, including any figure, date, or table heading. If you cannot quote the supporting text, you have not read the source closely enough to cite it yet. These phrases drive every later check. 2. **Decide how much capture the source earns.** Default for any source that carries a claim: two public captures, Wayback plus Archive.today. Add a local ArchiveBox copy when the claim is load-bearing, disputed, likely to be challenged, or about a page with a history of changing, such as a pricing page, a policy page, or a post by an interested party. A passing mention with no claim resting on it needs no capture. A source with a DOI or another persistent identifier gets cited by that identifier, with capture optional. 3. **Capture in the Wayback Machine first.** Use Save Page Now at https://web.archive.org/save or the direct form https://web.archive.org/save/. Save Page Now captures that one page, one time. Wait for the save to finish, then confirm the capture exists with the Availability API instead of trusting the save screen: Send a GET request to the Availability API endpoint at archive.org/wayback/available with two query parameters: url set to the page address, and timestamp set to the citation date in YYYYMMDD form. A ready-to-run curl form of both API calls is in `references/service-notes.md`. Record the capture URL and its 14-digit UTC timestamp from the response. An empty archived_snapshots object means the capture did not land. Retry once after a pause, then log it as failed and lean on the other services. A 429 response means rate limited: stop submitting, wait, and check again later through the Availability API. For batches, authenticated Save Page Now 2 saves use S3-style keys from an archive.org account. Keys are secrets and never appear in a sidecar, a script, or a log. 4. **Capture in Archive.today second.** Paste the same URL into the red save box at https://archive.today/ and submit. First check the blue box for an existing capture of the same URL, and reuse a same-day capture if one holds the cited text. While the save runs, read the element list the service shows during the crawl: each element with its size, status, and type. A list full of errors means a thin capture, whatever URL comes back. Record the snapshot URL when the save finishes. Keep batches small, since bulk saving can trigger repeated CAPTCHAs. 5. **Make the local ArchiveBox copy when step 2 called for one.** Add the URL to ArchiveBox and let the default extractors run. The record to check is singlefile.html inside the snapshot folder, with output.html as the fallback if SingleFile was off, screenshot.png for any claim about how the page looked, and the WARC file under warc/ as the replayable copy. Note in the sidecar if extractors were disabled, because a partial snapshot is a thinner record. Local means the copy survives even if both public services lose theirs, so the snapshot folder goes wherever the project keeps its records, with a backup. 6. **Run the completeness check on every capture. No exceptions.** A capture passes only if it holds the work, not just the address. Run the checker in this skill against each capture you can load as a file or fetch as a URL: `python3 scripts/archive_check.py --original-url --local-file --must-contain "" --must-contain "" --min-images --out .json` Or with `--capture-url ` in place of `--local-file`, and `--source-url` or `--source-file` to compare against the page as cited. The checker extracts visible text with scripts and styles stripped, tests every must-contain phrase, counts images, compares visible text length against the source when one is given (default floor: half), and hashes the capture bytes with SHA-256. Then do the two checks a script cannot do: open the capture and read the cited paragraph in place, in context, and look at the images the claim uses. A chart captured as a broken image icon fails even when the img tag is present. A capture that shows a login form, a consent wall, an error page, or an empty app shell fails whatever the status code said. 7. **Fix a failed capture the same day, while the live page still holds the text.** In order: try the other public service, try the Wayback save again after a pause, try ArchiveBox with a real browser session behind it for pages that need rendering, and for a page one service captured badly, check whether a capture from the other service on the same day holds the full text. If every capture fails and the live page is still good, save a local SingleFile or print-to-PDF copy yourself, hash it, and say plainly in the sidecar that this copy is self-made and carries less weight than an independent capture. If the live page has already changed or gone, go to step 9. 8. **Write the sidecar record for every cited source.** Start from `references/sidecar-template.json`. One record per source, filed next to the draft or in the project source folder, named for the source so it can be found without the draft. Required fields: original URL, the claim it supports in one sentence, the cited phrases, the date and time you read the live page, every capture with its service, capture URL or local path, capture timestamp, and completeness verdict, plus the SHA-256 and image count for any local file you checked. The hash proves the file you checked is the file you filed. It does not prove the page said anything, the phrase check does that. 9. **Rescue a citation whose page already changed or vanished.** Do not recapture the changed page and call it the source. Find the version that was live when the claim was written: - List captures around the citation date with the CDX API: send a GET request to the endpoint at web.archive.org/cdx/search/cdx with query parameters url set to the page address, output set to json, from and to set to the citation window in YYYYMMDD form, and fl set to timestamp,original,statuscode,mimetype,digest. A ready-to-run curl form is in `references/service-notes.md`. The digest field is the shortcut: rows with the same digest hold the same body, a new digest marks a change, so pick the last capture with the old digest, or the capture closest before the change. - Check Archive.today for a capture from the same window, since its captures are user submitted and often exist for pages the crawlers missed. - Verify the rescued capture with step 6, phrases first. A capture from the right week that lacks the cited words does not rescue anything. - Repair the citation: keep the original URL, add the verified capture as the archived-at link, and write the rescue into the sidecar: when the live page changed, what it says now or that it is gone, and which capture now carries the claim. The worked version of this, with rerunnable test files, is in `references/rescue-example.md`. 10. **Recheck the set before publication.** For each source in the draft: sidecar present, at least two captures recorded or a written reason there is only one, every linked capture passing the phrase check, and the archived-at link in the citation pointing at the capture that passed, not at a calendar page or a search page. Count the sources and report the counts plainly: how many cited, how many fully captured and checked, how many partial, how many failed. A draft with uncaptured claim-bearing sources is not ready, however good the prose is. ## Output Contract Every archiving job leaves: - Captures: at least two per claim-bearing source (Wayback and Archive.today by default, plus a local ArchiveBox snapshot where step 2 required one), each with its capture URL or local path and capture timestamp. - A completeness verdict per capture from `scripts/archive_check.py`, covering the cited phrases, image count, and visible text check, plus a human read of the cited paragraph in the capture and a look at the images the claim uses. - One sidecar record per source, following `references/sidecar-template.json`: original URL, claim, cited phrases, live page read date, all captures with verdicts, and SHA-256 for every checked local file. - Repaired citations where a page had changed: original URL kept, verified capture added as the archived-at link, and a rescue note in the sidecar stating what the live page does now and which capture carries the claim. - A closing count: sources cited, sources with two verified captures, sources with one, sources failed, each failed source named with the reason. ## Operating Rules - Capture at citation time. No archive service can capture a version it never saw, so a source archived after its page changed is archived in its changed form. The work in this skill happens while drafting, not after. - Never report a capture as good on the strength of the snapshot URL or a 200 status. The phrase and image check is the verdict, and the human read of the cited paragraph is part of it. - Never invent a capture URL, a timestamp, a hash, or a phrase. Capture URLs and timestamps come from the service response or the Availability or CDX API output. Phrases are quoted from the page, not paraphrased into the checker. - Never put archive.org S3 keys, or any other credential, in a sidecar, a skill file, a committed script, or a log. Authenticated saves read keys from the environment or the account store at run time. - Respect rate limits and CAPTCHAs as stop signs, not obstacles to route around. On a 429 or a CAPTCHA wall: pause, shrink the batch, and verify what already landed. Do not loop submissions. - Do not capture paywalled, login-only, or private pages into public archives. If a source is not public, cite it as a private source, keep a local copy only where you have the right to keep one, and say in the sidecar that no public capture exists. - A self-made local copy is a fallback, clearly labelled as self-made in the sidecar. It proves what you saved and when you hashed it. It does not carry the weight of an independent capture, and the citation notes should not pretend it does. - Keep the sidecar with the work. A capture set that lives only in a browser history or a chat thread is lost work. File sidecars in the project source folder, and keep local snapshots backed up with the project. - For academic or legal work, follow the venue rule where it is stricter: some journals and courts specify Perma.cc or a named archive. Use that service as one of the two captures, and keep the second capture and the sidecar anyway.