--- name: directory_accuracy_audit description: >- Audit a directory or listing database for dead entries: stratified sampling with finite population math, tool-neutral verification checks for any sector, ghost rate calculation with Wilson intervals, one evidence record per listing, a missing-entry reverse check, and a risk-based recheck cadence. Triggers when someone asks how accurate a directory is, wants ghost or dead listings found, needs an audit sample size, or needs a defensible accuracy report for a provider, business, service, or member directory. --- # Directory Accuracy Audit ## Purpose Directories rot quietly. Entities close, move, leave a network, or change a phone number, and the listing stays up, sending users to closed doors. A ghost listing is worse than a missing one in one specific way: the user trusts it, acts on it, and only learns the truth at the door, the dial tone, or the rejection letter. This skill audits any directory, in any sector, with any verification tools available. It does not depend on one calling platform or one industry database. The method is the product: a random stratified sample, the same verification protocol applied to every sampled listing, one evidence record per listing, and rates reported with confidence intervals so a reader can tell a measured problem from a guess. Two failure directions exist and both get measured. Ghosts and errors are listings that should not be there, or not like that. Missing entries are eligible entities that are not listed at all. A sample drawn from the directory cannot find the second kind, so the workflow includes a reverse check from an independent frame. ## Workflow 1. Freeze and snapshot. Export the full directory as the audit frame. Record the export date, the row count N, and who owns corrections. Save listed values exactly as published. All rates refer to this snapshot, not to whatever the live directory becomes during fieldwork. 2. Define the verdicts before sampling. Use the five verdicts in `references/verification-and-evidence.md`: verified, field_error, ghost (with subtype), duplicate, unverified. Write down which fields are critical for this directory: identity, existence, location, contact, status or inclusion claim, category and attributes. A listing that fails a critical field is not verified, even if everything else checks out. 3. Stratify. Split the frame by the variables most likely to carry different error rates: region, category or specialty, source feed, listing age, or owner. Every listing falls in exactly one stratum. Small strata that matter, a new feed, a region after a closure wave, get a minimum sample even if proportional allocation would give them almost none. 4. Size the sample. Run `scripts/audit_calc.py sample-size` with the population, a margin, and a confidence level. The script uses the Cochran proportion formula n0 = Z squared times p times (1 minus p), divided by margin squared, with p set to 0.5 by default for the conservative maximum, then applies the finite population correction n = n0 divided by (1 plus (n0 minus 1) divided by N). Defaults worth using: 95 percent confidence, margin 0.05 for a routine cycle, 0.03 when the result will drive removals or a public claim. If the required sample is most of the population, audit the lot and call it a census. 5. Allocate and draw. Run `scripts/audit_calc.py allocate` for proportional allocation across strata. Draw randomly inside each stratum with a recorded seed, and save the selected listing IDs before any checking starts. Hand-picked samples do not carry the confidence the math promises. Oversampled strata are fine, the scoring step weights them back by population. 6. Verify with one protocol for all. Apply the attempt protocol in the reference file identically to every sampled listing: automated checks, one independent source, direct contact attempts, then adjudication of ghost verdicts by a second reviewer. The independent source must not be the source the directory was built from. Checking a directory against its own feed proves nothing. 7. Record evidence per listing. One record per sampled listing with the listed snapshot, attempts, sources, verdict, wrong fields with listed and confirmed values, and the next action. Template and field list are in the reference file. No record, no verdict: a rate built on memory is not an audit. 8. Score. Tally per stratum into the CSV shape in `assets/example-results.csv` (stratum, population, sampled, verified, ghost, field_error, unverified, duplicate, with the counts summing to sampled) and run `scripts/audit_calc.py score`. See Output Contract for how the rates are defined. 9. Run the reverse check. Sample from an independent frame of entities that should be listed, check whether each is listed, and report the missing rate separately. Method in the reference file. 10. Act and schedule the recheck. Correct field errors, remove or merge ghosts and duplicates, carry unverified listings into the next cycle at higher priority, and set the cadence from the reference file: continuous automated checks, a quarterly sample audit as the default rhythm, triggered rechecks when a stratum degrades, and an annual full census of the high-harm slice. ## Output Contract The audit report contains: - Frame: N, export date, strata with population per stratum, sample size with the margin and confidence it was sized for, the random seed, and the protocol used. - Per stratum and overall: sampled, resolved (sampled minus unverified), ghost rate, any-error rate (ghost plus field_error plus duplicate, divided by resolved), unverified rate of sample, and estimated ghosts in the full population. - Ghost rate is ghosts divided by resolved listings, with a Wilson score 95 percent interval from `scripts/audit_calc.py score`. Unverified listings are excluded from that denominator and reported on their own line. Counting unverified as ghost inflates the rate and punishes listings for the audit team's failure to reach them. - Wilson intervals are used because ghost rates often sit near zero in healthy strata, where a normal approximation can produce intervals below zero and poor coverage. The script computes centre = (p plus z squared over 2n) divided by (1 plus z squared over n), with the matching Wilson margin, and clamps the result to 0 to 1. - Overall rates are weighted by stratum population, never a plain average of stratum rates. A small stratum with a bad rate must not outweigh a large clean one. - Missing rate from the reverse check, on its own line, with its own frame and sample described. - Action list: every ghost, duplicate, and field error from the sample, with owner and due date, plus the projected population counts labelled as estimates with their intervals. - Evidence file: one record per sampled listing, retained with the report. Worked example, run and verified with the bundled script: a directory of 12,000 listings in three strata (north 5,000, south 4,000, west 3,000). At 95 percent confidence and margin 0.03, sample-size returns n = 981 (unadjusted 1,067.11). Proportional allocation gives north 409, south 327, west 245. Scoring the tallies in `assets/example-results.csv` gives a weighted ghost rate of 11.32 percent, an estimated 1,357.8 ghosts in the population, and per-stratum ghost rates of 9.09 percent for north (Wilson 6.64 to 12.33 percent), 14.60 percent for south (11.13 to 18.93 percent), and 10.64 percent for west (7.31 to 15.23 percent). Read that report as: south is the problem stratum, the overall ghost problem is roughly one listing in nine, and the unverified rates near 3 to 4 percent are small enough that they do not change the story. ## Operating Rules - Snapshot before fieldwork, correct after the cycle locks, except a listing actively harming users, which gets corrected at once with its pre-correction snapshot logged. - Never count unverified as ghost, and never quietly drop unverified listings from the report. They are a finding about reachability and about audit effort. - Never extrapolate a rate from a convenience sample. If the sample was not random, report counts only and say the population rate is unknown. - Never verify a directory only against the source it was built from. At least one independent source or direct contact per sampled listing. - A ghost verdict needs adjudication: a second reviewer, or direct confirmation from the entity or an official registry. One unanswered phone call is an attempt outcome, not a verdict. - Report intervals with every rate. A ghost rate of 11 percent from 981 checks and the same rate from 40 checks are different claims, and the interval is what shows it. - Keep the missing rate and the ghost rate separate in every table and headline. Adding them produces a number that describes neither problem. - Sector rules can tighten this method, they do not replace it. One concrete floor: United States health plan directories fall under the No Surprises Act, which per the CMS toolkit requires verification at least every 90 days, updates within two business days of receiving a change, and a process to remove providers the plan cannot verify. Where a sector rule exists, cite the primary regulator document and follow the stricter of the rule and this method. - Evidence records carry personal data risk when listings are people. Keep contact-attempt notes factual and brief, store the evidence file with the directory's own access controls, and sample values in the public report stay aggregated. ## Quickstart ```bash python3 scripts/audit_calc.py sample-size --population 12000 --margin 0.03 python3 scripts/audit_calc.py allocate --population 12000 --sample 981 --strata "north=5000,south=4000,west=3000" # do the fieldwork, tally per stratum into the shape of assets/example-results.csv python3 scripts/audit_calc.py score --results assets/example-results.csv ``` For the verdict taxonomy, field checks by type, attempt protocol, evidence record fields, reverse check, and cadence rules, read `references/verification-and-evidence.md`.