---
name: "document_processing_pipeline"
description: "Build and run a complete document processing pipeline at scale: ingest and format conversion, OCR with accuracy tuning, PII redaction with verification checks, extraction QA, and archival output. Use when an agent must turn a folder of scans or mixed document files into searchable, redacted, archivable output, or when asked to design, tune, or audit a document pipeline."
---

# Document Processing Pipeline

## Purpose
Take messy input (scans, photos, PDFs, office docs, images) and produce output that is searchable, redacted of personal data, quality-checked, and archived in a durable format. Each stage has a gate: a document that fails a gate gets rerouted, retried, or flagged for human review. It never silently passes.

## Workflow
1. **Inventory and triage.** List every input. For each file, determine: has a native text layer or is image-only (try text extraction first; fewer than ~50 characters per page means image-only), page count, language, orientation.
2. **Normalize.** Deskew, de-rotate, normalize DPI to at least 300 for OCR targets. Keep originals untouched; normalized copies go to a working directory. Record the normalization parameters per file.
3. **OCR with tuning.** Do not run one default pass. Follow the tuning ladder in `references/ocr-tuning.md`: pick a page segmentation mode for the page layout, try a small set of preprocessing variants on the hardest pages, keep the variant with the best word-confidence score. Run the quality gate in `bin/ocr_gate.py`; pages below the gate get retried with different settings or flagged.
4. **Format conversion.** Convert output to a durable target: PDF/A (ISO 19005) for scans via OCRmyPDF with an embedded text layer and sidecar text, or direct text for native PDFs. See `references/archival-conversion.md`. Validate PDF/A claims with veraPDF, not with the tool that produced the file.
5. **PII redaction.** Define the identifier set up front (default: the HIPAA Safe Harbor 18 categories as a baseline detector set, even for non-health docs). Redact text layer AND image layer AND metadata. Then run `bin/redact_scan.py` as a verification pass: it rescans the output for identifier patterns and metadata leaks. Redaction is a two-pass discipline: the detector pass redacts, the verifier pass proves it. See `references/redaction-checklist.md`.
6. **Extraction verification.** Compare extracted text against a ground-truth sample: compute character error rate and word error rate on at least 10 representative pages. Check structural integrity (reading order, table rows, page counts match). Low-confidence word ratio above threshold means re-OCR, not hand edits.
7. **Package and report.** Write one run report: inputs, settings used, gate scores, pages retried, pages flagged, redaction findings, validation results. Output goes out only with the report attached.

## Output Contract
- Searchable archival files (PDF/A with embedded text layer), sidecar text, and a run report.
- The report states: per-file gate scores, which pages were retried and why, redaction detector set used and verifier results, ground-truth error rates, and a list of files needing human review.
- No file ships with an unverified redaction pass or a failing quality gate.

## Operating Rules
- Never invent accuracy numbers. Report measured confidence, measured error rates, and gate thresholds. "The OCR said so" is not verification.
- Confidence is a noisy signal, not a verdict. It guides retries and review, it does not certify a page as correct.
- Redaction covers every layer: text, image pixels (burn the boxes into the image, never a transparent overlay), and document metadata. Check all three.
- Keep the original files immutable. All processing writes to a working directory; the report links originals to outputs.
- Calibrate on your own documents. Publish no accuracy claim for a corpus you have not measured; thresholds come from your ground truth, not from a blog post.
- When a page fails every OCR variant, flag it for a human instead of shipping the best-bad output.