--- name: ai_screener_audit description: "Audit an AI or rules based hiring screener before it decides who gets seen: build a test set from real past applications, measure adverse impact and score drift across groups, check reasons for sensitive term and proxy leakage, and test whether a human can really override it. Use when a screener, resume ranker, chatbot screen, or assessment score is about to launch, has changed, or needs a bias audit, a Local Law 144 style review, or a remediation plan." --- # AI Screener Audit ## Purpose Give the owner of a hiring screener a verdict they can act on, backed by their own outcome data. A quick harness run against fixtures can show a tool runs. It cannot show who the tool is screening out in real use, whether it drifted after a vendor update, whether its stated reasons leak proxies for protected traits, or whether the human review step is real. This audit covers all four, then turns each finding into a fix with an owner and a recheck date. This is an audit workflow, not legal advice. Where a finding points at discrimination law, say so plainly and route the decision through counsel. ## Workflow 1. Scope and freeze. Record tool name, vendor, exact version, threshold or ranking rule, the jobs and locations it feeds, and the decision it influences: screen out, rank, shortlist, or invite. Freeze the version for the audit period. If the vendor updates the tool mid audit, split the data at the update and audit both sides. A result averaged across two versions describes neither. 2. Check independence and authority. The auditor must not have built, sold, or operated the tool, and must be able to publish a bad result. Under the NYC Local Law 144 rules, an auditor is not independent if they were involved in using, developing, or distributing the tool, worked for the employer or vendor during the audit, or hold a direct or material indirect financial interest in either. Write the independence statement into the report before looking at results. 3. Build the test set from real past applications. Use the most recent complete period with enough volume, usually the last two to four quarters. Include rejected applicants, not just hires. Outcomes for hires alone cannot show who was screened out. Keep group data separate from the scoring run, joined only for analysis, and report how many rows have an unknown group. Drop no group silently. 4. Treat historical labels with suspicion. Past hiring decisions carry past bias. A model that matches historical hires well may be matching the bias, not the job. Report accuracy against historical outcomes as agreement with the past, never as proof of fairness or job relatedness. Ask separately what evidence links each major input to actual job performance: a job analysis, a content or criterion validation study, or nothing. Nothing is a finding. 5. Run adverse impact on real outcomes. Run `bin/audit.py` once per dimension on an applicant level CSV with columns group, selected, and optional score. It prints selection rates, impact ratios, a two proportion z test against the highest rate group, and small sample warnings. - Selection rate: the share of a group selected to move forward. - Scoring rate, for scored tools: the share of a group scoring above the median score of the whole sample. - Impact ratio: a group rate divided by the highest rate in that table. The reference is the highest rate group in the data, not a group picked in advance. - Run three tables: sex, race and ethnicity, and the intersection of the two. Single dimension tables hide harm that lands on one intersection, as the worked example shows. - Categories under 2 percent of the data may be left out of the ratio call under Local Law 144 practice, but their count and rate still get reported and the exclusion gets a written reason. 6. Read the numbers with both tests, never one. The four fifths rule from the Uniform Guidelines, 29 CFR 1607.4D, flags a selection rate below 80 percent of the highest rate as evidence of adverse impact. It is a screening rule of thumb, not a finding of discrimination, and clearing it is not proof of lawfulness. Pair every flagged ratio with the significance result and the group size. Significance testing has low power on small samples, so a large gap on a small group is a reason to collect more data and look harder, not a pass. A tiny gap can be statistically significant on a huge sample and still matter little in practice. Report the gap size, the ratio, and the p value together. 7. Measure drift, not just the current snapshot. Recompute the same tables for the prior period, or for the period before the last vendor change, using the same method. Compare impact ratios and score distributions per group. Drift patterns to name in the report: - Stable: ratios and distributions hold across periods. - Shifted, no outcome change: scores moved but selection outcomes did not. Watch it, shorten the next interval. - Shifted with outcome change: a group ratio fell after a version, threshold, prompt, or feature change. That change is the prime suspect. Name it, date it, and require the vendor changelog entry. 8. Hunt leakage in features and reasons. Sample at least 100 stated reasons or explanations, plus the full feature list, and search for protected terms and close proxies. Check at minimum: postal code, name, school, graduation year or other age proxies, employment gaps, commute distance, language or dialect markers, and prior employer prestige. Illinois law makes the point concrete: using zip codes as a proxy for a protected class is itself a violation there. A reason that reads clean can still rest on a dirty feature, so trace any flagged phrase back to the input that produced it. Quote each leak in the report with a count. 9. Test the human override for real. Do not accept a policy document as evidence. Plant seeded cases: at least 20 borderline and clearly qualified applicants the tool scored low, run them through the normal review queue, and record how many a reviewer overturns, how long each review takes, and whether the override changes the recorded outcome. Check the live override rate by group for the period. Near zero overrides, reviews measured in seconds, or overrides that leave the outcome unchanged all mean the same thing: the human step is a rubber stamp. Effective oversight means the reviewer understands the output, has authority to reverse it, and can stop the tool, which is the standard the EU AI Act Article 14 sets for high risk systems, and hiring systems are listed as high risk in its Annex III. 10. Check notice and audit duties by location. Examples, to confirm with counsel for the actual sites involved: NYC Local Law 144, enforced since 5 July 2023, requires an independent bias audit within one year before use, a published summary, and candidate notice at least 10 business days before use, with penalties up to $500 for a first violation and $500 to $1,500 for each later one. Illinois HB 3773, in force since 1 January 2026, bans AI use that has a discriminatory effect, bans zip code proxies, and requires notice, but does not itself require a bias audit. Federal Title VII, the ADA, and the ADEA apply to automated selection regardless of any local audit law, and the employer stays responsible for a vendor tool used on its behalf. 11. Write findings as fixes. Use `references/report-template.md`. Every finding gets evidence, severity, a named fix, an owner, a date, and a recheck method. Fix options in order of preference: remove or replace the leaking feature, pick an alternative procedure with equal validity and less impact, which the Uniform Guidelines at 29 CFR 1607.3B and 1607.6 direct where one exists, adjust the threshold with documented validity cost, retrain or revalidate, or pause the tool for the affected job. Retraining is the last resort, not the default. It is slow and it can bake the same labels back in. 12. Decide and date the next audit. End with one decision: keep, keep with fixes on a deadline, pause for the affected job or group, or stop. Set the next audit date at signing. Annual at the latest where Local Law 144 applies, and always after a material change to model, threshold, features, or job. ## Worked Example: A Screener That Drifted Illustrative example with invented applicant counts. The arithmetic is real and can be checked with `bin/audit.py`. A resume screener, version 2.2, closed its baseline quarter with a race table that passed the screening flag: white applicants 200 in, 80 selected, 40.0 percent. Black applicants 180 in, 58 selected, 32.2 percent. Impact ratio 0.81. The vendor shipped version 2.3 mid year. No changelog entry mentioned scoring changes. The next quarter, same jobs, same threshold on paper: - White: 210 in, 88 selected, 41.9 percent, ratio 1.00 - Asian: 100 in, 40 selected, 40.0 percent, ratio 0.96 - Hispanic: 150 in, 45 selected, 30.0 percent, ratio 0.72, flagged - Black: 190 in, 48 selected, 25.3 percent, ratio 0.60, flagged The intersection table was worse. Black women: 95 in, 19 selected, 20.0 percent. White men, the highest rate intersection: 110 in, 50 selected, 45.5 percent. Ratio 0.44. The single dimension sex table showed a milder gap and would have understated the harm if the audit had stopped there. Leakage sampling of 120 reasons found 17 that cited career continuity, traced to an employment gap feature, and 9 that cited school names in a way that tracked a prestige list in the feature set. Seeded override testing planted 24 qualified low scored applicants. Reviewers overturned 1, with a median review time under 15 seconds. The override existed on paper only. Fixes assigned: remove the gap and school prestige features and rerun the quarter as a counterfactual, owner named, two weeks. Pause version 2.3 for the two affected jobs until the counterfactual table clears 0.80 or a validated alternative is in place. Require a vendor changelog with scoring impact notes before any future update goes live. Recheck in 30 days on live data, full re-audit in 90 days. Decision: pause, not retrain. The labels had not changed, the features and the silent update had. ## Output Contract Deliver one audit report using `references/report-template.md`, containing: scope and frozen version, auditor independence statement, data period and row counts including unknowns, outcome tables for sex, race and ethnicity, and intersections with ratios and significance, a drift comparison against the prior period, leakage findings with quoted examples and counts, the seeded override test result, job relatedness evidence and its gaps, findings as fixes with owners and dates, and a keep, fix, pause, or stop decision with the next audit date. Also deliver the CSV extract and the `bin/audit.py` output for each table, so any reader can recompute every number in the report. ## Operating Rules - Real outcomes or no verdict. Fixture runs and vendor demo data support a smoke test only. The audit verdict rests on real applications, including rejections. - Never report an impact ratio without its group size. A ratio on 12 applicants and a ratio on 12,000 applicants are different kinds of evidence. - Never clear a tool because a vendor says it was tested. Ask what data, what groups, what period, and what ratios came out. No numbers means no test, for audit purposes. - Score drift plus outcome drift after a silent vendor change is a stop and ask event. Freeze the version, keep the data, and make the vendor explain the change in writing. - Reasons are evidence. If a stated reason would embarrass the owner read aloud to the rejected applicant, treat it as a finding until traced and cleared. - An override that cannot change the outcome is not oversight. Report it as absent. - Passing four fifths is not a clean bill of health. Other laws, other groups, and intersectional harm can still be present. Say what was tested and what was not. - Keep group data used for auditing separate from group data used for deciding. The audit needs protected attributes to measure harm. The screener must not use them to score. - This workflow follows the measurement logic in the NIST AI Risk Management Framework, Measure and Manage functions: measure with stated metrics, manage with owned mitigations and a monitoring cadence. It does not certify compliance with any law.