document_processing_pipeline v1.0.0

name: "document_processing_pipeline" description: "Build and run a complete document processing pipeline at scale: ingest and format conversion, OCR with accuracy tuning, PII redaction with verification checks, extraction QA, and archival output. Use when an agent must turn a folder of scans or mixed document files into searchable, redacted, archivable output, or when asked to design, tune, or audit a document pipeline."

Document Processing Pipeline

Purpose

Take messy input (scans, photos, PDFs, office docs, images) and produce output that is searchable, redacted of personal data, quality-checked, and archived in a durable format. Each stage has a gate: a document that fails a gate gets rerouted, retried, or flagged for human review. It never silently passes.

Workflow

1. Inventory and triage. List every input. For each file, determine: has a native text layer or is image-only (try text extraction first; fewer than ~50 characters per page means image-only), page count, language, orientation.
2. Normalize. Deskew, de-rotate, normalize DPI to at least 300 for OCR targets. Keep originals untouched; normalized copies go to a working directory. Record the normalization parameters per file.
3. OCR with tuning. Do not run one default pass. Follow the tuning ladder in references/ocr-tuning.md: pick a page segmentation mode for the page layout, try a small set of preprocessing variants on the hardest pages, keep the variant with the best word-confidence score. Run the quality gate in bin/ocr_gate.py; pages below the gate get retried with different settings or flagged.
4. Format conversion. Convert output to a durable target: PDF/A (ISO 19005) for scans via OCRmyPDF with an embedded text layer and sidecar text, or direct text for native PDFs. See references/archival-conversion.md. Validate PDF/A claims with veraPDF, not with the tool that produced the file.
5. PII redaction. Define the identifier set up front (default: the HIPAA Safe Harbor 18 categories as a baseline detector set, even for non-health docs). Redact text layer AND image layer AND metadata. Then run bin/redact_scan.py as a verification pass: it rescans the output for identifier patterns and metadata leaks. Redaction is a two-pass discipline: the detector pass redacts, the verifier pass proves it. See references/redaction-checklist.md.
6. Extraction verification. Compare extracted text against a ground-truth sample: compute character error rate and word error rate on at least 10 representative pages. Check structural integrity (reading order, table rows, page counts match). Low-confidence word ratio above threshold means re-OCR, not hand edits.
7. Package and report. Write one run report: inputs, settings used, gate scores, pages retried, pages flagged, redaction findings, validation results. Output goes out only with the report attached.

Output Contract

• Searchable archival files (PDF/A with embedded text layer), sidecar text, and a run report.
• The report states: per-file gate scores, which pages were retried and why, redaction detector set used and verifier results, ground-truth error rates, and a list of files needing human review.
• No file ships with an unverified redaction pass or a failing quality gate.

Operating Rules

• Never invent accuracy numbers. Report measured confidence, measured error rates, and gate thresholds. "The OCR said so" is not verification.
• Confidence is a noisy signal, not a verdict. It guides retries and review, it does not certify a page as correct.
• Redaction covers every layer: text, image pixels (burn the boxes into the image, never a transparent overlay), and document metadata. Check all three.
• Keep the original files immutable. All processing writes to a working directory; the report links originals to outputs.
• Calibrate on your own documents. Publish no accuracy claim for a corpus you have not measured; thresholds come from your ground truth, not from a blog post.
• When a page fails every OCR variant, flag it for a human instead of shipping the best-bad output.