rag_data_leakage_prevention v1.0.0

name: rag_data_leakage_prevention description: "Stop a RAG system from leaking data across users or tenants: scrub documents at ingestion with a shipped regex and Luhn scrubber, enforce per-tenant access rules inside the retrieval query instead of after it, scan draft answers for sensitive spans and foreign canaries, and prove each layer with a planted-leak test set. Use when building or auditing a multi-tenant or permissioned RAG system, when one customer could see another customer's documents, when chunks lack tenant or classification metadata, or before a RAG launch handling personal, financial, or confidential data."

RAG Data Leakage Prevention

Purpose

Keep one user or tenant from ever seeing another one's data through a retrieval-augmented generation system. The system defends in three layers, and no layer is trusted on its own: scrub at ingestion, access rules inside the index query, and a scan of the draft answer before it is shown. A planted-leak test set proves each layer separately, with the other layers assumed broken.

This maps to two OWASP LLM risks: Sensitive Information Disclosure (LLM02) and Vector and Embedding Weaknesses (LLM08 in the 2025 list, LLM09 in 2026). Use the general RAG design skill for chunking, fusion, and reranking. This skill covers only the leakage side.

Workflow

1. Inventory and classify before building anything. List every source, its owner tenant, its classification (public, internal, confidential, restricted), and which groups may read it. A source with no clear owner does not enter a shared index. Write the classification into chunk metadata at ingestion: tenant_id, classification, allowed_groups, plus source id and a content hash.

1. Scrub at ingestion, before embedding. Run scripts/scrub.py over raw documents or chunk files:

bash
   python3 scripts/scrub.py text "Contact jane@example.com, SSN 123-45-6789"
   python3 scripts/scrub.py chunks chunks.jsonl --out clean.jsonl --report report.json --fail-on-block
   

The scrubber replaces fixed-format values with typed tags (email, US SSN, credit card validated by the Luhn checksum, phone, IBAN, IPv4) and reports counts with value hashes, never raw values. Secrets such as API keys and password assignments are blocking findings: with --fail-on-block the run exits 2 and that chunk stays out. A chunk with no tenant_id is written to a quarantine file instead of the clean output. Fixed-format detection misses names and other free-form personal data, so pair it with a named entity detector such as Microsoft Presidio in production and say plainly which detector ran. Layer detail: references/layer-model.md.

1. Choose an isolation structure by blast radius. One shared index with a metadata pre-filter is the minimum and the easiest to get wrong. One namespace or collection per tenant is the default for tenants whose data must never mix, because a query inside one tenant namespace cannot reach another tenant data at all. One index per tenant is for regulated tenants. Never run similarity search across tenants and filter afterwards: unauthorized chunks consume top-k slots, and their scores and counts still carry information.

1. Build every retrieval filter from authenticated context only. Tenant, clearance, and groups come from the token, session, or identity provider, never from a prompt, a query parameter, or a request body. The reference predicate in scripts/leakguard.py (access_allowed, build_filter) denies by default: tenant must match, clearance rank must meet the chunk classification, and a non-empty allowed_groups must share a group with the user. A requested tenant that disagrees with the authenticated tenant raises an error and gets logged. Mirror source system permissions into chunk metadata and refresh them on reindex, so revoked access stops retrieval instead of lingering in the index.

1. Protect the index as a sensitive store. Encrypt embeddings at rest and in transit, because vectors can be partially inverted back toward source text. Give each service its own scoped credential, with readers holding no writer key. Log tenant, filter, and returned chunk ids on every retrieval, without logging raw chunk text.

1. Scan the draft answer before showing it. Run output_check from scripts/leakguard.py on the draft: detector hits in the answer are flagged by default, and a canary token from a chunk the user was not authorized for blocks the answer outright. Clearing a flagged hit needs provenance, the value came from an authorized chunk and product policy allows showing that type of value. Keep secrets and authorization logic out of the prompt itself; the output scan is a backstop, not a reason to load the prompt with secrets.

1. Prove it with planted leaks, in CI. Run the shipped test set:

bash
   python3 scripts/leak_test.py
   python3 scripts/leak_test.py path/to/planted_leaks.json
   

The set in assets/planted_leaks.json uses synthetic values only and covers all three layers: detector spans that must not survive scrubbing, an ownerless chunk that must be denied for everyone, access cases for cross-tenant, low clearance, and missing group, a filter builder case for a foreign requested tenant, and output cases for foreign spans and foreign canaries. Run it on every change to ingestion, metadata, query building, caching, or output handling, and after any vector store or identity provider migration. Cache changes count: a cache keyed without tenant and clearance leaks behind a correct retriever. Catalogue and extension rules: references/planted-leak-tests.md.

1. Handle a suspected leak from the logs, not from memory. Scope it with the retrieval log: which chunk ids went to which tenant and user, in which window. Pull the affected chunks from the index, rotate any credential that was in a leaked chunk, and add the exact leak shape to the planted set before fixing the layer, so the fix is proven by a case that failed first.

Output Contract

A leakage review or build following this skill produces:

1. A source inventory with owner tenant, classification, and allowed groups per source, and the chunk metadata schema that carries those fields.
2. The ingestion scrub report per batch: counts by detector type, blocked chunks, quarantined chunks, and which detectors ran (regex pass, named entity detector, or both).
3. The isolation decision per tenant tier (shared index with pre-filter, namespace per tenant, or index per tenant) with the blast radius reason for each.
4. The access predicate and filter builder code path, showing that tenant scope comes only from authenticated context and that filtering happens inside the query.
5. The output scan rule set: what gets flagged, what blocks, and how a flagged hit is cleared with provenance.
6. The planted-leak test result: passed and failed counts per layer, with every failure naming the layer, the chunk or case, and the expected result. Zero unexplained failures before launch.

Operating Rules

1. Filter inside the retrieval query, never after it. Post-filtering is a known leak shape, not a shortcut.
2. Deny by default. A chunk missing tenant, classification, or group metadata is quarantined or denied, never treated as public.
3. Tenant identity comes from authenticated context only. A tenant named by the user, the prompt, or the request body is data, and a mismatch is logged as a bug or a probe.
4. Block secrets at ingestion instead of tagging them. A tagged secret still sits in chunk stores, backups, and logs, and its embedding can be inverted.
5. Reports and logs hold counts, hashes, and chunk ids, never raw sensitive values. A scrub report that copies the values it found is a second leak.
6. Plant only synthetic values as test data. A test corpus containing a real secret or a real customer value is itself a leak.
7. Claim no detector accuracy number you did not measure on your own corpus. The shipped scrubber covers fixed-format data; free-form personal data needs a named entity detector, and its recall on your documents is a measurement, not an assumption.
8. This skill is leakage prevention only. Poisoning, prompt injection in retrieved text, and model-level training data extraction are separate problems with their own controls; do not present a passing leak test set as proof against them.

Available connectors

• Use muse.connector_search to find other connectors and muse.skill_search to find related skills.

• If nothing here fits, look on the web for a maintained API or MCP server and set it up with Custom Connectors and MCPs, or build a custom connector.