systematic_review_screening v1.0.0

name: systematic_review_screening description: "Screen literature for a systematic review: design testable inclusion and exclusion criteria, run a piloted dual independent title-abstract and full-text workflow, resolve conflicts, and measure inter-rater agreement with Cohen kappa. Use when screening records for a systematic, scoping, or rapid review, writing eligibility criteria, calibrating reviewers, building a PRISMA flow count, or reporting screening reliability."

Systematic Review Screening

Purpose

Turn a review question into screening decisions another reviewer can reproduce. The skill covers criteria design, pilot calibration, dual independent screening at title-abstract and full-text stage, conflict resolution, agreement metrics, and the counts a PRISMA 2020 flow needs. It does not run the search and it does not extract data or assess risk of bias. Those are separate steps with their own methods.

Workflow

1. Frame the question before writing criteria

Pick the framework that fits the question type, then write the question in one sentence.

• Intervention or effectiveness: PICO (Population, Intervention, Comparison, Outcome), or PICOS if study design is a real filter.
• Qualitative or mixed methods: SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type).
• Scoping review: PCC (Population, Concept, Context).

If a comparator has to be invented to make PICO fit, the framework is wrong. Switch frameworks instead of forcing the question.

2. Write criteria a stranger could apply

For each framework element, write three things: the include rule, the exclude rule, and a boundary case with its ruling. A criterion that needs the author in the room to interpret it is not finished.

Rules for usable criteria:

• One decision per criterion. Split combined rules such as "adults with the condition treated in primary care" into population, condition, and setting, so a disagreement points at one element.
• Define features, not labels alone. "Randomised trial" is a label. Add what counts when the abstract does not use the word: allocation described, control arm present, and so on.
• Cover report characteristics as well as study characteristics: date range, language, publication status, minimum follow up if it matters. PRISMA 2020 item 5 asks for both, plus a rationale for any notable restriction.
• State the outcome rule plainly. Say whether a study that measures none of the review outcomes is out, or in for other outcomes. Avoid vague wording such as "no relevant data".
• Order criteria so cheap checks come first at full text: population, then design, then intervention or phenomenon, then outcome. The first failed criterion becomes the primary exclusion reason.
• Write the criteria into the protocol before screening starts. If a rule changes mid review, date the change, give the reason, and record how already screened records were handled. That goes in the final report as a protocol deviation.

Use assets/criteria_worksheet.md to draft the set. A filled example is in references/worked-example.md.

3. Pilot and calibrate

Never start full screening on untested criteria.

1. Both reviewers independently screen the same pilot sample. Pick the sample in the protocol: a fixed count or a share of the records, large enough to include a few likely includes and the boundary cases the team argues about.
2. Compute agreement with bin/agreement.py and read the disagreement list, not just the number.
3. Classify each disagreement: unclear wording, missing information in the abstract, or a genuine judgement split. Only the first class is fixed by rewriting a criterion. Add the ruling to the worksheet as a new boundary case.
4. Re-pilot on a fresh sample if wording changed in a way that could flip earlier decisions.
5. Record the pilot outcome in the screening log: sample size, kappa, what changed.

A working target many teams use is to keep calibrating until disagreements are mostly judgement splits on hard cases rather than two readings of the same sentence. Treat any kappa threshold the team sets as its own protocol choice.

4. Screen title and abstract, dual and independent

• Deduplicate first, into one library, and keep the duplicate count for the flow diagram.
• Two reviewers screen each record independently. Neither sees the other decision until both are in.
• Decisions: include, exclude, maybe. At this stage maybe means "cannot tell from title and abstract", and maybe goes forward. Err toward inclusion: a record wrongly excluded here is gone, a record wrongly included gets caught at full text.
• Record exclusion reasons at this stage only if the tool makes it cheap. Cochrane guidance notes reasons are usually not recorded at title and abstract stage. They are required at full text.
• Unblind, list conflicts, resolve by discussion. Bring in a third reviewer only for conflicts discussion does not settle.

5. Screen full text, with one primary reason

• Get the full text for every record that survived stage 4, and link multiple reports of the same study so the study is counted once.
• Two reviewers independently apply the same criteria. For each exclusion assign exactly one primary reason, the first criterion in the worksheet order that the study fails.
• Keep a separate list of near miss studies a reader would expect to see, with their reasons. PRISMA reporting expects the excluded studies and reasons to be visible.
• Resolve conflicts the same way as stage 4, and log the final decision, who resolved it, and how.

6. Measure and report agreement

Run the calculator on the pre-resolution decisions. Agreement measured after consensus is meaningless, it will always be perfect.

bash
python3 bin/agreement.py decisions.csv --collapse-maybe include --out agreement-report.md
python3 bin/agreement.py --both-include 18 --both-exclude 62 --r1-only 8 --r2-only 12
python3 bin/agreement.py pilot.csv --json

Report all of these, for each stage separately:

• Records scored by both reviewers, agreements, disagreements
• Percent agreement and Cohen kappa, with the marginal counts per reviewer
• How disagreements were resolved
• PABAK as well when inclusion is rare, so readers can see the prevalence effect

Interpretation help and the formula are in references/agreement-metrics.md. The Landis and Koch labels are a convention for discussion, not a pass mark.

7. Close out with PRISMA counts

Produce the flow numbers: records identified, duplicates removed, records screened, records excluded at title and abstract, reports sought, reports not retrieved, reports assessed, reports excluded with reasons grouped by primary reason, studies included, reports of included studies. PRISMA 2020 item 8 also requires the report to state how many reviewers screened each record and report, whether they worked independently, and any automation tools used.

Output Contract

Deliver, every time:

1. Criteria worksheet, filled, with boundary cases and rulings, plus a dated change log if any rule moved after the protocol.
2. Screening log CSV in the shape of assets/screening_log_template.csv: one row per record, both reviewers decisions per stage, final decision, primary exclusion reason at full text, resolver.
3. Agreement report per stage: n, percent agreement, kappa, marginals, disagreement list, and how each class of disagreement was handled.
4. PRISMA flow counts as a table, with full text exclusion reasons grouped and counted.
5. A short methods paragraph ready to paste into a manuscript, stating criteria source, reviewer count, independence, pilot, resolution method, and agreement results.

Operating Rules

• Accuracy first: every method claim in the write up must trace to PRISMA 2020, a Cochrane or Campbell handbook chapter, or the review protocol itself. If a number or threshold came from team preference, label it as a team choice.
• Never screen alone and call it dual screening. If only one reviewer is available, say so, screen a sample in duplicate after a time gap if that is the plan, and report the limitation plainly.
• Never compute agreement on post consensus decisions, on a sample chosen because it looked easy, or after deleting the maybe category without saying how maybes were collapsed.
• Never quietly tighten or loosen a criterion mid review. Change, date, re-pilot, and re-screen the affected records, or report the deviation.
• Do not exclude at full text for a reason that is not in the criteria worksheet. If a new reason shows up repeatedly, that is a criteria change, handle it as one.
• Keep automation honest: if a tool ranked, suggested, or auto excluded records, name the tool, its role, and any human check on its decisions, as PRISMA 2020 item 8 requires.
• Style: plain sentences, no em dashes in any output.

References

• references/agreement-metrics.md: kappa formula, a hand checked worked calculation, the prevalence problem, when to use Fleiss kappa or PABAK.
• references/worked-example.md: a complete small review carried from question to flow counts.
• assets/criteria_worksheet.md: blank criteria worksheet with boundary case slots.
• assets/screening_log_template.csv: the log shape the calculator and the flow counts expect.