citation_integrity_audit v1.0.0

---

name: citation_integrity_audit description: "Audit citations for claim support: trace each claim to a locator in its source, detect hallucinated, chimeric, anchorless, and retracted citations, and separate format checks from support checks. Use when reviewing a draft, paper, report, brief, or AI-generated text before publication, when a citation looks plausible but unverified, or when someone asks whether the references actually back the claims."

---

Citation Integrity Audit

Purpose

Check that every load-bearing claim in a document is backed by a real source, at a findable location in that source, saying what the document says it says. A citation can fail in four separate ways, and a clean result on one says nothing about the others:

1. Existence: the source is real and the identifier resolves.
2. Identity: the resolved source is the one cited (same title, authors, venue, year).
3. Support: the source actually states the claim, at the cited location.
4. Status: the source has not been retracted, corrected in a way that changes the claim, or superseded.

Formatting is a fifth, separate check. A perfectly formatted citation to a paper that does not exist passes format and fails everything that matters.

Workflow

1. Build the claim ledger first

Before checking any source, extract the claims that need support. A claim needs a ledger row if it states a number, a date, a causal effect, a quote, a finding attributed to a study, or a fact a reader could reasonably doubt. Opinion, framing, and common knowledge agreed with the reader do not.

Copy assets/claim-ledger.csv and fill one row per claim: claim_id, claim_text, citation as written, doi_or_url, locator, verdict (leave blank until step 4), evidence_note.

Run the extractor to catch identifiers and quotes you missed:

python3 bin/citation_audit.py extract draft.md

It lists DOIs with a syntax check, URLs, and long quotes. Each long quote needs a locator in its citation or it starts the audit as anchorless.

2. Existence and identity, registry first

For each citation with a DOI, check the registry before opening anything else:

python3 bin/citation_audit.py check-doi 10.xxxx/xxxxx --online

The tool queries https://api.crossref.org/works/{doi}, the metadata registry publishers deposit into. Three outcomes:

• FOUND: compare every returned field against the citation as written. Title, first author, container, and year must all match. One field matching and others differing is the chimera pattern: real parts assembled into a reference that never existed as written. Record the mismatched fields.
• NOT_FOUND (404): the DOI is not registered. That alone does not prove fabrication, since a real paper can carry a mistyped DOI. Search the exact title in Crossref, OpenAlex (https://api.openalex.org/works), and the publisher site. No title match anywhere, plus no author record of the work, is what earns a NOT_FOUND verdict for the citation.
• UNVERIFIABLE: network or registry error. Retry once. If it still fails, mark UNVERIFIABLE and say so in the report. Never convert a failed check into a pass.

For citations with no DOI, search the exact title in quotation marks in at least two scholarly databases (Crossref, OpenAlex, PubMed for biomedical work, the publisher catalog for books). A title search always returns something. A near match is not the cited paper. Compare title, authors, and year before treating any hit as the source.

Warning that matters: never ask the tool or model that produced a citation whether the citation is real. In Mata v. Avianca, 678 F. Supp. 3d 443 (S.D.N.Y. 2023), attorneys filed a brief citing judicial opinions that did not exist, and the record shows the generating tool had assured them the cases were real. The court imposed a $5,000 sanction. Verification means an independent registry or database, not the generator's own answer.

3. Claim tracing: the locator test

Existence is not support. For each surviving citation, trace the claim:

1. State the claim in one sentence, exactly as the document makes it. Note qualifiers: population, dose, timeframe, direction, and whether the document states it as certain or tentative.
2. Find the passage in the source that is supposed to carry it. Record the locator: page, paragraph, section heading, table or figure number, or timestamp. If you cannot name a locator, the verdict cannot be SUPPORTED.
3. Read that passage, plus enough context to catch a reversed condition (for example a result that held only in a subgroup, or a correlation the document upgraded to a cause).
4. Compare claim to passage on five points: number, direction, population, certainty, and scope. A mismatch on any one is a finding, and the finding names which point failed.

Direct quotations get the strictest form of this test: the quoted words must appear at the locator, word for word. Under APA 7th, a direct quote requires a locator (page, or paragraph, section, or timestamp when there are no pages). A quote with no locator, or a locator where the words do not appear, is a finding even if the source is real.

Anchorless citations: a citation attached to a paragraph of several claims, with no locator and no clear mapping to one claim, gets traced claim by claim. If no single passage carries a given claim, that claim is UNSUPPORTED even when the source is on-topic.

4. Status check

For each source that passes steps 2 and 3, check whether its status changed after publication:

• Crossref record: look for update-to entries (correction, retraction) in the metadata, which the check-doi output prints.
• OpenAlex record: check the is_retracted flag.
• Publisher page: look for a retraction notice, expression of concern, or correction linked from the article. COPE retraction guidance expects retracted articles to be clearly marked and linked to their notice, so an unmarked publisher page plus a registry flag still counts as RETRACTED: trust the flag, note both.
• Legal citations: confirm the case exists in a court or legal database and has not been vacated or reversed on the point cited. A case name plus a plausible reporter citation is exactly the shape of the Mata fabrications, so treat unverifiable cases as NOT_FOUND, not as obscure.

A retracted source cited for its findings is a finding regardless of how well it supports the claim. Citing a retracted paper to discuss the retraction itself is legitimate and should be worded that way in the document.

5. Secondary source check

Flag any claim the document cites through an intermediary. Signals: "as cited in", a citation whose passage you can only find quoted inside another paper, or a famous old finding cited to a recent review.

Rule from APA 7th practice: cite the primary source directly when you can get it. Use the secondary form sparingly, name both in text with "as cited in", and list only the secondary source in the reference list. In the audit, a claim supported only by a secondary source gets at most PARTIAL until the primary is opened, because secondary paraphrases drift.

6. Format check, last and separate

Only after support verdicts are set, check format against the required style (APA, MLA, Chicago, IEEE, Vancouver, or the venue's own): author order and punctuation, year placement, title case, container, volume and pages, DOI presented as a https://doi.org/ link, reference list ordering, and in-text to list matching (every in-text citation has a list entry, every list entry is cited).

Report format errors in their own section. Never let a format fix change a support verdict, and never present a document as verified because its formatting is clean.

7. Verdicts and report

Assign one verdict per ledger row:

• SUPPORTED: source real, identity matches, passage at the recorded locator states the claim on all five comparison points.
• PARTIAL: source supports a weaker or narrower version (smaller effect, subgroup only, tentative wording the document stated as fact). Report the version the source does support.
• UNSUPPORTED: source is real but no passage carries the claim.
• CONTRADICTED: source states the opposite or a materially different number.
• NOT_FOUND: no registry or database record after the step 2 searches.
• CHIMERA: fields split across two or more real works, or title and identifier point to different works.
• RETRACTED: source was retracted or withdrawn.
• UNVERIFIABLE: checks could not be completed (paywall with no abstract or library route, registry outage). State what was tried.
• FORMAT_ONLY: only format was checked for this row, because the client asked for a format pass. Never mix these rows into a support summary.

Validate the finished ledger, then report:

python3 bin/citation_audit.py ledger claim-ledger.csv

The validator rejects a SUPPORTED row with no locator or no evidence note, duplicate claim ids, and verdicts outside the set above.

Report format: counts by verdict, then one entry per non-SUPPORTED row with claim, citation, finding, the evidence (where you looked and what you found), and the fix (correct source, corrected wording, or delete the claim). Close with what was not checked and why. Full method detail and worked audits are in references/worked-audits.md and references/failure-taxonomy.md.

Output Contract

Deliverables for a full audit:

1. The completed claim ledger (CSV, schema in assets/claim-ledger.csv), validated by the ledger command.
2. A findings report: verdict counts, one entry per problem claim, and an unchecked-items section.
3. For each fixable problem, a concrete fix: the correct citation, the corrected claim wording the source does support, or a clear delete recommendation.

Sampling rule: for documents with more than 50 citations, a sampled audit is acceptable only if the client agrees to sampling in advance. Audit all citations carrying numbers, quotes, and causal claims, then a random sample of the rest, and state the sample size and selection method in the report. Never describe a sampled audit as a full audit.

Operating Rules

1. Every factual claim in the audit report itself follows the same standard: a verdict rests on a registry response, a database record, or a passage you opened, and the evidence_note says which.
2. A search result snippet is not a source passage. Open the source or its abstract before assigning SUPPORTED.
3. Do not silently repair citations. A corrected DOI, year, or author is a reported finding plus a fix, because the error itself is evidence about how the document was produced.
4. Do not invent locators. If a source has no page numbers, use paragraph, section, or timestamp. If none exists, say the source is unpaginated and quote the heading you used.
5. Paywalled sources: check the abstract, registry metadata, and any lawful open copy. If the claim depends on body text you cannot reach, the verdict is UNVERIFIABLE, not SUPPORTED.
6. Bulk similarity is a warning sign, not a verdict. Many citations in identical phrasing, journals that do not appear in any directory, or volume numbers that do not match the cited year all justify a closer look, and none of them alone proves fabrication.
7. Keep the audit trail: save the ledger and report with the document version they describe. Citation findings against a moving draft are worthless if nobody can tell which draft they belong to.