--- name: "reviewer_verdict_adjudication" description: "Decide whether an automated PR reviewer actually passed a change: poll each reviewer to an explicit positive completion marker for the current head commit, never infer a pass from silence or an empty findings list, dig findings out of collapsed sections, uncounted inline comments, check annotations, and separate threads, triage every finding, log every unearned pass in a fail-open ledger, and merge only on an earned verdict. Use when a PR has bot or AI reviewers (CodeRabbit, Copilot, or any check or review bot), when auto-merge or a human is about to merge on a bot's word, or when auditing past merges for passes nobody earned." --- # Reviewer Verdict Adjudication ## Purpose Automated reviewers fail quiet. A bot that is still running, timed out, reviewed an old commit, or hid its findings inside a collapsed section looks the same as a bot that passed, if all you check is that nothing red is showing. This skill turns "the bot seems fine with it" into an earned verdict per reviewer, per head commit, with the evidence quoted. It works for any automated reviewer on GitHub style pull requests: check bots, summary bots, and review bots, named or custom. It does not review the code itself. Use a code review checklist for that. This skill decides whether the reviewers you already rely on actually finished and actually passed. ## Workflow ### 1. List the expected reviewers before polling - Write down every reviewer this PR is supposed to get, and how each one reports: a check run, a submitted review, a summary or walkthrough comment, inline comments, or a mix. - Write down, for each reviewer, the exact positive completion marker you will accept. Examples: check status `completed`, a finished walkthrough comment present for the current commit, the words Review completed, or an APPROVED review tied to the head SHA. - If you cannot name a marker for a reviewer, that reviewer cannot earn a pass in this run. Mark it UNEARNED and say why. - Record the current head SHA first. Every later read must match it. GitHub requires status checks to pass on the latest commit, and a review or check from an earlier commit does not satisfy a required check. ### 2. Poll to the marker, not to silence - Poll each reviewer with a bounded loop and a timeout your team sets. Read real state each round, for example `gh pr checks `, `gh pr view --json headRefOid,reviewDecision,statusCheckRollup,latestReviews,reviews,comments`, and the review comments endpoint for inline threads. - Treat these states as not finished: check status `queued`, `in_progress`, `pending`, `requested`, `waiting`, or `expected`, and review state `PENDING`. `gh pr checks` uses exit code 8 for pending, so a script must handle that code instead of reading it as failure or success. - Stop polling only on one of three outcomes: the completion marker appears, a blocking result appears, or the timeout hits. Timeout is UNEARNED, never a pass. - Re-read the head SHA after polling. If a push landed during the poll, restart for the new head. A stale approval can be dismissed automatically on push when that branch rule is on, and even when it is not, the old verdict describes old code. ### 3. Read the verdict the reviewer actually gave - For a check run, require status `completed` and read the conclusion. On GitHub, `success`, `neutral`, and `skipped` are the passing conclusions. `failure`, `action_required`, `timed_out`, `cancelled`, `stale`, and `startup_failure` block. - Watch the skipped trap: a skipped required check can count as passing on GitHub, so a skip earns a pass only if you read why it skipped and the skip reason is valid for this change, for example a job level condition on files this PR does not touch. - For a submitted review, read the state: `APPROVED`, `CHANGES_REQUESTED`, `COMMENTED`, `DISMISSED`, or `PENDING`. `CHANGES_REQUESTED` blocks. `DISMISSED` earns nothing. `COMMENTED` is not an approval. - Know each reviewer's ceiling before counting it. GitHub Copilot, by default, leaves a Comment review, not Approve or Request changes, so by default it does not count toward required approvals. Mark comment only reviewers as advisory in the snapshot and pair them with a gating check or a human approval. Some repos change that config, so verify the config in the repo instead of assuming. ### 4. Dig findings out of the hiding places - Expand every collapsed section. Some reviewers wrap their walkthrough in a collapsible details block by default, and findings and file lists sit inside it. An unexpanded walkthrough is an unread walkthrough. - Count inline comments from the review comments data, not from the conversation counter. The conversation view folds, paginates, and groups threads, so its visible count can miss comments. - List separate review threads, resolved and unresolved. Check outdated threads against the current diff instead of assuming the push fixed them. - Check the PR description as well as the comments. Some reviewers write or edit a summary into the description itself. - Read check run annotations and the check details page, not just the icon. A completed check can carry findings in its annotations while the summary icon looks clean. - Fetch the next page when a comments or reviews page is full. Zero findings on page one of two is not zero findings. ### 5. Triage every finding, then adjudicate - Give every finding one disposition: fixed, wontfix with a reason a human accepted, false positive with evidence, or duplicate of a named finding. Open blocking findings keep the verdict at BLOCKED. - Save the evidence as a snapshot JSON and run the bundled adjudicator from the skill folder: ``` python3 scripts/adjudicate.py examples/worked_snapshot.json python3 scripts/adjudicate.py snapshot.json --ledger fail-open-ledger.csv ``` - The adjudicator returns EARNED_PASS, BLOCKED, or UNEARNED per reviewer and MERGE_OK only when every reviewer earned a pass. It checks marker seen, check completed, commit current, blocking findings zero, and the three hiding place flags set to true. - Copy the checklist in `references/checklist-and-ledger.md` for the per PR record, including the worked adjudication for PR 512 (earned) and PR 513 (unearned and blocked). ### 6. Log every unearned pass - When any system or person was ready to merge on a verdict the adjudicator calls UNEARNED or BLOCKED, append a row to the fail-open ledger, even if the merge was stopped in time. The adjudicator writes the row for you with `--ledger`. - Review the ledger weekly. Repeated rows for one reviewer or one cause (silence, green icon only, stale approval, zero comment count, skipped check) mean fix the setup: make the reviewer a required status check, turn on dismissal of stale approvals, or stop counting a comment only reviewer as a gate. ## Output Contract Deliver one adjudication note per PR with: - PR number, repo, and head SHA adjudicated - A table or list per reviewer: name, completion marker quoted, check status and conclusion, review state and commit, findings total, blocking open count, hiding places checked, and verdict (EARNED_PASS, BLOCKED, or UNEARNED) with reasons - Overall verdict: MERGE_OK or DO_NOT_MERGE, matching the adjudicator output - The snapshot JSON path and, if any verdict was not earned, the ledger rows added - What would change the verdict, stated as concrete next steps (wait for a named marker, fix a named finding, re-run on the new head) ## Operating Rules - Never infer a pass from silence, an empty findings list, a zero comment count, a green icon, or a reviewer not showing up. Absence of findings is evidence only after a completed review for the current head has been read in full. - Never merge on a verdict for a different commit than the one being merged. - Never let the author of the change, or the agent that wrote it, be the only adjudicator of its reviewers. The adjudication note must quote markers and states a second reader can recheck in under a minute. - A timeout, a crashed poll, a rate limited read, or a missing page is UNEARNED. Say which read failed. Do not fill the gap with a guess and do not retry a rate limited provider in a loop. - Overrides are decisions, not verdicts. If a human merges over a BLOCKED or UNEARNED verdict, log it in the ledger with the reason and the approver name, and keep the original verdict in the note unchanged. - Keep claims tied to reads from this PR. Platform defaults cited in this skill (check conclusions, review states, Copilot comment only default, collapsed walkthrough default) come from GitHub and CodeRabbit docs, and repo config can override several of them, so verify the config that applies to the repo in front of you.