prompt_provenance_audit v1.0.0

name: "prompt_provenance_audit" description: "Audit what a shipped AI product is actually told: collect dated system prompts and tool schemas, label each artifact as vendor published, vendor reported, captured, archived copy, behavior inferred, or unverified, hash and diff versions, and write a report that never upgrades evidence or guesses past the artifacts. Triggers when someone asks what instructions a product runs on, whether a leaked prompt is real or current, how a prompt changed between versions, or wants a prompt or tool list checked against a vendor claim." metadata: { "includeInPrompt": true }

Prompt Provenance Audit

Purpose

Find out what instructions and tools a shipped AI product was given, using dated artifacts, and report only what those artifacts support.

The core problem is provenance. A prompt text can be exact vendor text for one surface on one date, a vendor description of intended behavior, a capture someone pulled from a live product, a copy of a copy in a public archive, or a guess built from outputs. Those five things look alike as plain text and support very different claims. This skill keeps them separate, from collection through the final report.

Two anchor examples, both verified from primary sources:

• Anthropic publishes the exact system prompts for Claude on claude.ai and its mobile apps in dated release notes, and states plainly that those updates do not apply to the Claude API. Exact text, narrow scope.
• OpenAI publishes a Model Spec that describes intended behavior and a chain of command, and states that the public version may not include every detail and that production models do not yet fully reflect it. Authoritative about intent, not a transcript of a live system prompt.

Workflow

1. Scope before you collect. Write down the product, the surface (web, mobile, API, custom agent, embedded assistant), the model name or version if known, and the date window. Every later claim inherits this scope. A prompt for the web surface says nothing on its own about the API surface.
2. Collect vendor material first. Check the vendor's own docs, release notes, help center, and official repository for that product and surface. Save the page or file exactly as retrieved, with its URL and the retrieval date. Note any scope limit the vendor states on the page, word for word if it is short.
3. Collect captured and archived material second. Public archives of extracted prompts exist and are useful starting points. OWASP's system prompt leakage page links two of them as references, the LouisShark chatgpt_system_prompt collection and the Jujumilk3 leaked-system-prompts collection. Treat anything from an archive as an archived copy until you can trace it to an original capture with a date, a surface, a model version, and a method. Record who captured it and how, if the archive says. If it does not say, that gap goes in the report.
4. Capture your own only where you are allowed to. On a product you own or are authorized to test, you may record what the product discloses and save the full exchange as an artifact. Tools such as garak, the NVIDIA LLM vulnerability scanner, can probe your own system for data leakage in a structured way. Do not present a capture from one account tier, region, or date as the product's prompt in general.
5. Freeze every artifact. Save the raw text untouched. Run the manifest command for each file and record its hash in the report: python3 scripts/provenance_audit.py manifest <file> --label <label> --source-url <url> --claimed-date <date> --surface <surface> --model <model> The hash lets a reader prove later that the text you diffed is the text you collected. If an artifact changes under you, the hash changes, and that is a finding, not a nuisance.
6. Label every artifact, and never upgrade a label. Use the label rules in the next section. A label can only move down in strength when new facts appear, for example when a supposed vendor text turns out to be an archive copy. It never moves up because the text looks right, reads fluently, or matches what a model says about itself.
7. Normalize only a copy, only for diffing. To compare two versions, make a working copy and apply one stated normalization: trim trailing spaces, unify line endings, sort nothing, rewrite nothing. Keep the raw files as collected. State the normalization in the report, because a diff of normalized copies can hide a real change in whitespace or ordering.
8. Diff the text. Run python3 scripts/provenance_audit.py diff <old> <new> and read the full diff yourself. Group the changes under plain headings: identity and role, behavior rules, refusal and safety wording, formatting rules, knowledge or date statements, tool instructions. Quote the exact added and removed lines for any change you report.
9. Diff the tool schemas separately. Tool lists are structured data, so compare them as data, not as prose. Run python3 scripts/provenance_audit.py tools <old.json> <new.json>. Report tools added, tools removed, parameters added or removed, and required changes. Then cross check: a prompt rule naming a tool the schema does not offer, or a schema tool no prompt rule mentions, usually means the two artifacts came from different builds or surfaces. Say so as a question for the vendor or the archive, not as a settled fact.
10. Separate prompt findings from behavior findings. A line vanishing from a prompt does not prove the behavior vanished. Behavior also comes from training and from controls outside the prompt. OWASP makes the related point that people using a product can work out many of its guardrails from outputs whether or not they ever see the exact wording. If you test behavior, save the test exchanges as their own dated artifacts with the label captured, and keep the two evidence sets apart in the report.
11. Write the report in the contract format below. Include a "does not claim" section every time. If a key fact cannot be tied to an artifact, either cut it or move it to that section as an open question.

Label rules

Use exactly one label per artifact:

• vendor_published: the vendor hosts the exact text on its own domain or official repository, with a date or version. Supports claims about that text, that surface, and that date.
• vendor_reported: the vendor describes the instructions or behavior in its own words but does not publish the exact live text. A model spec, a blog post, and a help page summary all land here. Supports claims about stated intent. Does not support quoting a live prompt.
• captured: someone recorded the text or exchange from the live product, with date, surface, model version, and method all recorded. Supports claims about what that session disclosed on that date.
• archived_copy: the text sits in a third party archive or a repost. Its strength depends entirely on the capture it points back to. If it points back to nothing, treat its claims as unverified while keeping the label honest.
• behavior_inferred: a conclusion drawn from outputs, with no prompt text behind it. Never quote behavior inferred material as if it were prompt text. Write it as an observation about outputs.
• unverified: no date, no surface, no method, or no traceable source. Report its existence if it matters. Build no finding on it.

Three tests before a label sticks:

• Can you open the source yourself right now?
• Does the source state its own date, surface, and scope?
• Can a second person retrieve the same artifact and get the same hash?

A model agreeing that a leaked text matches its instructions is behavior, not a vendor record. It cannot raise a label. Screenshots without a retrievable source stay captured at best, and unverified if the date or surface is missing.

Output Contract

A finished audit delivers a report with these parts, in this order:

• Scope: product, surface, model or version, date window, and the exact question the audit answers.
• Artifact table: one row per artifact with file name, label, source URL, claimed date, retrieval date, SHA-256, word count, and headings, all taken from the manifest output.
• Text findings: grouped changes, each one backed by a quoted added or removed line and the two artifacts it came from.
• Tool findings: tools added, removed, and changed, from the tools diff, plus any prompt and schema mismatch stated as an open question.
• Behavior findings, if any: kept in their own section, tied to their own captured test artifacts.
• Does not claim: surfaces not covered, dates not covered, questions the artifacts cannot answer, and any artifact whose provenance chain has a gap.
• Method note: normalization applied, commands run, and where the raw artifacts are stored.

A complete small example, with real command output on fictional files, is in references/worked-audit.md.

Operating Rules

1. Accuracy rule. Every factual claim in the report must trace to an artifact in the table or to a primary source you cite. Never invent dates, versions, quotes, or provenance. If you cannot verify it, cut it or list it under "does not claim."
2. Scope discipline. Never carry a finding across surfaces, models, or dates without an artifact for the new scope. The Anthropic case is the standing warning: vendor published text for web and mobile, with the vendor itself stating it does not apply to the API.
3. Prompts are not secrets and not security controls. OWASP LLM07:2025 states this directly. In your own product audits, flag credentials, connection strings, internal host names, and permission rules found in prompt text as exposure findings, and recommend moving enforcement outside the model. Do not treat prompt wording itself as a control that can hold a boundary.
4. No motive guessing. A diff shows what changed. It does not show why. Do not write that a vendor added a rule in response to an incident, a competitor, or a news event unless a cited vendor source says so.
5. Verbatim means verbatim. Quoted prompt lines must match the frozen artifact byte for byte apart from the stated normalization. If you shorten a line, mark the cut. Paraphrase must be labeled as paraphrase.
6. Handle captured material with care. Captured and archived prompts are other parties' product text. Use them for audit, comparison, and reporting. Do not republish a full captured prompt as your own product, and do not use an audit as a route to pull credentials or private data out of a product you do not own or lack authorization to test.
7. Freshness check. Prompts change without announcement on most products. Before reusing an older audit, re-retrieve the vendor source, rerun the manifest, and compare hashes. A changed hash means the old findings are history, not current fact.