paper_collection_management v1.0.0

name: "paper_collection_management" description: "Build a paper collection from a messy list: deduplicate by DOI, other identifiers, and title plus author plus year, check file integrity with SHA-256 and PDF header checks, and produce an auditable manifest with a missing item report. Use when cleaning a bibliography, assembling a literature review corpus, auditing a folder of PDFs, or reporting what was found and what is still missing." metadata: { "includeInPrompt": true }

Paper Collection Management

Purpose

Turn a messy list of papers and a folder of files into a collection a second person can audit. The point is provenance discipline: every unique paper is counted once, every duplicate merge has a stated reason, every file has a measured status and hash, and every missing or corrupt item is listed instead of hidden. A batch download with no manifest cannot be trusted, checked, or finished by someone else.

Use this skill for literature review corpora, reading list cleanups, lab reference folders, systematic review identification stages, and any handover where the question is what do we actually have.

Workflow

1. Freeze the input

• Save the messy list exactly as received, with its original filename and date. Work on a copy, never on the only original.
• Confirm the list format. The bundled tool reads CSV with a header row or JSON lines, with columns title, authors, year, doi, url, arxiv, pmid, file. Map other formats into those columns first and note the mapping in the audit note.
• Count input rows before any cleaning. That count is the denominator every later count must reconcile against.

2. Normalize for matching only

• Normalize DOIs by stripping a resolver prefix or a leading doi label, stripping surrounding punctuation, and lowercasing for comparison. Keep the supplied form in provenance and the normalized form in doi_norm.
• Normalize titles by applying Unicode NFKD, dropping combining marks, lowercasing, and collapsing punctuation and whitespace to single spaces. Keep the display title untouched.
• Extract the first author surname for matching. If the name is Family, Given, the surname is before the comma. Otherwise it is the last token.
• Read references/dedupe-rules.md before any merge if the list is large, contested, or part of a formal review.

3. Deduplicate in tiers, merging not deleting

• Tier 1: merge rows with the same normalized DOI.
• Tier 2: merge rows with the same normalized arXiv ID or the same PMID.
• Tier 3: merge rows with the same normalized title, same first author surname, and same year. Title and year alone never merges.
• Tier 4: flag high similarity title pairs as candidates and stop. A person confirms or rejects each pair, with the decision and reason recorded. Never auto merge a fuzzy pair, because a false merge removes a distinct paper without a trace while a missed duplicate only costs a later manual catch.
• Keep every source row number on the keeper record. Pick a keeper that has a DOI and a file when one exists, and record the tier that caused the merge in dedupe_method.
• Keep a preprint and its journal version as separate rows linked by a note, unless the project protocol written in advance says to treat them as one work.

Run the tiers with the bundled tool:

python3 scripts/paper_collection.py build --input messy.csv --files-dir ./pdfs --out-dir ./collection

4. Verify an identity against Crossref when it matters

• When a DOI is present but the title or year looks wrong, or when a key paper has no DOI, check Crossref before guessing. Use the REST API at https://api.crossref.org/works/{doi} for a single DOI, identify the client with a mailto parameter to use the polite pool, and treat the result as deposited publisher metadata that can be incomplete or later corrected.
• If Crossref and the list disagree on title, keep the Crossref title as a note, keep the list title in the audit note, and flag the row for a human call. Do not silently rewrite a bibliography from one API response.
• If no DOI can be found from the publisher page, Crossref, or a library catalogue, leave doi_norm empty and mark the paper as without DOI in the summary. An empty field is honest, a guessed DOI is a false record.

5. Check every file

• For each file named in the list, check existence, size, SHA-256 hash, and for PDFs the %PDF- header in the first bytes and the %%EOF marker near the end. Flag a file that starts with HTML as corrupt, it is usually a landing page saved as a PDF.
• Assign exactly one status per unique paper: present_ok, present_corrupt, missing, or not_expected. Definitions and limits are in references/integrity-checks.md.
• Spot check content by hand. Open a sample of present_ok files and confirm the first page matches the manifest title. The integrity checks prove a file is a complete PDF, they do not prove it is the right paper.
• Group manifest rows by sha256 as an extra audit. The same hash under two different DOIs means the same file was filed twice under two names.

Check one file or re verify a finished manifest with:

python3 scripts/paper_collection.py check-file ./pdfs/paper.pdf
python3 scripts/paper_collection.py verify --manifest ./collection/manifest.csv

6. Build and audit the manifest

• Produce manifest.csv, manifest.json, missing.csv, candidates.csv, and summary.json in one run. The schema for each column and file is in references/manifest-schema.md.
• Check the reconciliation before anything is shared: input_rows equals unique_papers plus duplicates_removed, and the four file status counts sum to unique_papers. If either check fails, the run is wrong, fix it and rebuild.
• Write the audit note beside the manifest: input source and date, command used, files directory checked, any Crossref lookups with dates and outcomes, every candidate decision with a name and reason, and any rule exceptions.

7. Report what is missing and finish retrieval honestly

• Hand over missing.csv as the work list. Each row there needs a next action: retrieve from the DOI, request through a library, ask the author, or accept as metadata only with a stated reason.
• Only retrieve full text the project has a right to access, from an open access copy, a publisher page, a library, or an author copy. Record the source of each new file, add it, and rebuild the manifest so the new hash and status are measured, not typed in by hand.
• If a paper cannot be obtained, leave it as missing in the final manifest and say so in the report. A collection with honest gaps is usable, a collection that hides gaps is not.

Output Contract

• manifest.csv and manifest.json with one row per unique paper, using the columns in references/manifest-schema.md, including doi_norm, file_status, sha256, source_rows, and dedupe_method.
• missing.csv listing every missing and corrupt item, as the retrieval work list.
• candidates.csv listing every fuzzy pair that needs a human decision, or an explicit statement in the report that there were none.
• summary.json and a report that states all counts: input rows, unique papers, duplicates removed by tier if known, present_ok, present_corrupt, missing, not_expected, with DOI, without DOI, and candidate pairs needing review.
• An audit note with input provenance, tool command, lookup log, candidate decisions, and exceptions.
• No claim about a paper, file, or count that is not in the manifest or the audit note.

Operating Rules

1. Every factual claim must be verified via web search or a primary source. Never invent numbers, dates, quotes, or events. Unverifiable = cut it or frame as opinion. No confident unverified claims, ever.
2. Normalization is for matching only. Never overwrite display titles, author names, or the original list with normalized forms.
3. Fuzzy similarity never merges on its own. It produces a candidate, a person decides, and the decision is recorded.
4. Merge duplicates, do not delete them. Merging keeps source rows, collections, tags, and citation links that deletion would lose.
5. A hash proves a file matches its recorded hash. It does not prove the file is the correct paper or the version of record, say so when reporting.
6. Counts must reconcile before the manifest is shared. If they do not, report the failure instead of adjusting a count by hand.
7. Do not bypass a paywall, login, or access restriction to fill a missing row. Mark it missing, record why, and use a lawful route.
8. Keep the manifest versioned when it is shared or cited. Do not renumber stable local IDs after sharing.

References

• references/dedupe-rules.md for tier definitions, normalization details, and what counts as a distinct paper.
• references/manifest-schema.md for column definitions, summary reconciliation, and audit note contents.
• references/integrity-checks.md for file status definitions, PDF checks, and re verification.
• scripts/paper_collection.py for the build, verify, and check-file commands. No network calls, no third party packages.
• examples/messy.csv for a small input showing an exact DOI duplicate written two ways.