pii_redaction_pack v1.0.0

name: pii_redaction_pack description: Redact PII from documents and data files before sharing: validated detection for emails, SSNs, cards, IBANs, phones and context-gated birth dates, four replacement strategies (replace, mask, keyed hash, consistent pseudonym), and a verification pass that catches metadata, tracked changes, and hidden parts. Use when someone asks to redact, scrub, anonymize, de-identify, or clean a document, spreadsheet, log, or export before sending it out, or to check whether a shared file still leaks personal data.

PII Redaction Pack

Purpose

Make a file safe to hand to someone else, and prove it. The pack ships a tested stdlib-only scanner and redactor (bin/redact.py) plus a verification checklist (references/verification-checklist.md) that covers the places redaction usually fails: metadata, tracked changes, comments, embedded objects, and black-box-over-text PDF fakes.

Detection is pattern plus validation, not guesswork:

• Credit cards: 13 to 19 digits, separators allowed, must pass the Luhn check digit test. A digit run that fails Luhn is not flagged as a card.
• US SSNs: must pass SSA assignment rules. Areas 000, 666, and 900 to 999, group 00, and serial 0000 are rejected, so order numbers in SSN shape stay alone.
• IBANs: must pass the mod-97 check.
• Birth dates: flagged only next to a context word (born, dob, date of birth). A bare date is usually a deadline, not an identity.
• Names and street addresses have no reliable pattern. Supply them as a known-literal list (--names) and the tool finds every case-insensitive occurrence. Add house patterns (account numbers, case IDs) as regex lines with --extra.
• For name detection at scale in free text, Microsoft Presidio adds NER-based PERSON and LOCATION recognizers on top of the same pattern approach. Use it when the literal list is not enough, and still run the verification pass after.

Workflow

1. Scope first. Ask who receives the file and what they actually need from it. Redact to that need. A support ticket needs the error and the timeline, rarely the customer's name, card, or birth date.
2. Scan the original. python3 bin/redact.py scan input.txt --names names.txt prints counts, line numbers, and masked previews as JSON. Exit code 2 means findings. Reviews show previews only, never full values, so the scan report itself does not become a leak.
3. Pick a strategy (table below) and redact to a new file. The tool refuses to overwrite its input. python3 bin/redact.py redact input.txt -o output.txt --strategy pseudonym --names names.txt
4. Strip the container. For office files, PDFs, and images, body redaction is half the job. Follow the metadata and hidden-content sections of references/verification-checklist.md (ExifTool plus qpdf re-linearize for PDFs, accept tracked changes and clear document properties for office files).
5. Verify. python3 bin/redact.py verify output.txt --names names.txt must exit 0. For docx, xlsx, pptx, odt, pdf, and images, verify also scans the container parts and returns metadata warnings with exit code 2. Then work the rest of the checklist by eye.
6. Ship only the verified output. Keep the original private. Keep the scan report, which holds counts and types only, with your own records.

Replacement strategies, following the options NIST SP 800-188 lists for removing direct identifiers:

| Strategy | Output looks like | Use when | |---|---|---| | replace | [REDACTED:EMAIL] | Default. Recipient needs the shape of the document, not the values. | | mask | --0147, card keeps last 4 | Support and billing talks where the last 4 confirm identity. | | hash | [HASH:EMAIL:58645683fdf9f92d] | Joining records across files without sharing values. Keyed HMAC-SHA-256, key in PII_HASH_KEY. The tool refuses a bare unkeyed hash, which is easy to reverse for low-entropy values like SSNs. | | pseudonym | [EMAIL-1], [KNOWN_LITERAL-1] | Case notes and research text where the same person appears many times and the thread must stay readable. Consistent within one run. |

Worked example, tested end to end. Input line: SSN 078-05-1120. Card 4111 1111 1111 1111. The scan flags both (078-05-1120 passes SSA rules, the card passes Luhn). Pseudonym redaction gives SSN [US_SSN-1]. Card [CREDIT_CARD-1]. A lookalike line, Order total 1234 5678 9012 3456, is left untouched because that number fails Luhn.

Output Contract

• The redacted file, at a new path, never overwriting the source.
• A scan or verify report as JSON: counts by type, line numbers, masked previews, metadata warnings. No full detected values in any report.
• A short note to the user naming the strategy used, the counts removed, the verification result, and any checklist item that could not be checked (for example, a PDF that needs a tool not installed).

Operating Rules

• Never send, upload, or share the unredacted original while "checking" something. All work happens on local copies.
• Black rectangles over text are not redaction. In a PDF, delete the underlying text or rasterize the page, then verify by extracting text and scanning it.
• Masking keeps part of a value readable by design. Use it only when the recipient has a real reason to see the last 4. Otherwise replace or pseudonymize.
• Quasi-identifiers count. ZIP code, birth date, and sex in combination can single out a person with no name present. If a data file keeps all three, say so plainly and let the owner decide.
• Hashing without a key is not protection for short values. The tool enforces a key; do not work around it with a plain SHA-256 of an SSN.
• Automated detection misses things: names written as initials, values spelled out in words, handwriting in scans. The eye pass in the checklist is mandatory, not optional polish.
• If verification still finds PII and you cannot remove it with the tools at hand, stop and report what remains and where. Never ship a file described as redacted when the verify pass failed.
• This pack reduces sharing risk. It does not by itself make a dataset de-identified under HIPAA, GDPR, or any other rule, and it never gets described that way.