meta_analysis v1.0.0

name: meta_analysis description: "Pool results across studies with a meta-analysis: write and register a protocol before searching, choose the effect size by outcome type, pool with fixed or random effects, read heterogeneity with Q, I2 and tau2 together, check small study effects honestly, and write results that state their limits. Use when combining effect estimates from several studies, asked for a pooled effect, a forest or funnel plot, a heterogeneity reading, or a meta-analysis methods section. Ships a tested calculator for effect sizes, DerSimonian-Laird, REML and Paule-Mandel pooling, Hartung-Knapp intervals, prediction intervals, Egger test and leave-one-out."

Meta-Analysis

Purpose

Combine effect estimates from several studies into one pooled estimate whose uncertainty is stated honestly. The skill covers the full run: protocol first, effect size choice by outcome type, model choice, heterogeneity, small study effects, plots, and a write-up with a real limits section. It does not run the literature search or screen records. Screening has its own method; use it before this skill starts.

The bundled calculator is bin/meta.py. It is standard library only and its self-test checks known values for pooling, Q, Hedges g, an odds ratio, and REML on homogeneous data.

Workflow

1. Write the protocol before searching

• Fill in assets/protocol_template.md: question and framework, eligibility, full search strings per source, data items, effect measure per outcome, model, tau2 estimator, interval method, pre-specified subgroups, sensitivity analyses, and the conditions under which pooling will be refused.
• Register or timestamp the protocol before data extraction where the field has a register (PROSPERO for health reviews), and log every later change as a dated amendment with a reason.
• Lock the model choice here, in writing. A model picked after seeing both pooled p values is a result, not a method.

2. Choose the effect size by outcome type

Pick one measure per outcome and pool on the scale where the sampling distribution is close to normal:

• Same continuous scale in every study: mean difference, pooled in original units.
• Continuous outcome on different scales: standardised mean difference, reported as Hedges g. g applies the small sample correction J = 1 - 3 / (4 df - 1) to Cohen d, with df = n1 + n2 - 2 and variance of g equal to J squared times the variance of d. In the write-up, translate the pooled g back into a familiar unit or a plain size statement.
• Binary outcome: odds ratio or risk ratio, pooled on the log scale and back transformed for reporting. Pick OR or RR in the protocol, not per study. For zero cells the calculator adds 0.5 to all four cells and flags it in the output; report that you did it.
• Correlation: Fisher z, variance 1 / (n - 3), pooled on the z scale and back transformed with tanh.
• Precomputed effects: supply yi and vi (or sei) directly, and name the measure in the protocol so readers know what scale the pooling used.

Never pool raw proportions, raw mean differences across different scales, or a mix of adjusted and unadjusted estimates without a pre-specified rule for which estimate wins per study.

3. Pool, with the model the question needs

Run the calculator:

bash
python3 bin/meta.py studies.csv --tau2 REML --out report.md --json result.json --forest forest.svg --funnel funnel.svg
python3 bin/meta.py --selftest

Input CSV shapes, one row per study, study column required: yi,vi or yi,sei; or m1,sd1,n1,m2,sd2,n2 with --measure SMD or MD; or e1,n1,e0,n0 with --measure OR or RR; or r,n.

• Common effect (fixed) model: weights 1 / vi, one true effect assumed. Defensible when studies are near replications of each other and the question really is about that one effect.
• Random effects model: weights 1 / (vi + tau2), true effects vary across studies and the pooled number is their mean. This is the usual case when studies differ in population, dose, setting, or follow-up.
• Tau2 estimator: REML as the default. DerSimonian-Laird (--tau2 DL) is the older closed form estimator and reads low when heterogeneity is real and studies are few; Paule-Mandel (--tau2 PM) is a good sensitivity check. Report all estimators you ran, and say which one the headline uses.
• Report the Hartung-Knapp interval alongside the conventional interval, and lead with it when k is small. It rescales the pooled standard error by the observed scatter and uses a t distribution on k - 1 degrees of freedom, so it prices in that tau2 was estimated. Simulation work (Partlett and Riley, Statistics in Medicine 2017) found it held coverage well in most scenarios, with slightly low coverage when heterogeneity was low and study sizes varied a lot.
• Add the prediction interval whenever tau2 is above zero: pooled mean plus or minus t on k - 2 degrees of freedom times sqrt(tau2 + SE squared). It answers a different question from the confidence interval: where the effect in a new study or a new setting is expected to fall.

4. Read heterogeneity so it can change the conclusion

Report all three, every time:

• Cochran Q with its degrees of freedom (k - 1) and p value. Q tests whether the scatter exceeds sampling error; with few studies it has low power, and with many precise studies it flags trivial scatter.
• I2, the share of total variation that sits beyond sampling error. Cochrane Handbook rough guides overlap on purpose: 0 to 40% might not be important, 30 to 60% may be moderate, 50 to 90% may be substantial, 75 to 100% considerable. Read it with the size and direction of effects, never as a pass or fail grade.
• Tau2 and tau, the absolute spread of true effects on the effect scale. I2 is a proportion and can read high when studies are precise but barely differ, or low when studies are noisy but truly differ. Tau is the number that feeds the prediction interval and the reader's judgement.

Then act on what you found:

• Effects scattered but pointing the same way: pool with random effects, report the prediction interval prominently, and investigate pre-specified subgroups or meta-regression.
• Effects in genuinely opposite directions, or one clear outlier with a study level reason: present results study by study, or pool with and without the outlier as a sensitivity analysis. A single average of effects that cancel is a number nobody experienced.
• No heterogeneity detected with few small studies: say detection was weak. Absence of a signal in a low power test is not proof of homogeneity.

5. Check small study effects without overclaiming

• Look at the funnel plot first: effect against standard error, pooled mean as the centre line. Read shape, not just symmetry: a missing corner, a gap, or small studies stacked on one side.
• Run the Egger test only when there are at least 10 studies and a real spread of study sizes. The calculator prints a warning below that floor. With similar sized studies the regression intercept is an extrapolation and means little.
• For binary outcomes pooled as odds ratios, Egger's test has a built-in problem: the log odds ratio and its standard error are mathematically linked, so asymmetry can appear with no bias at all. Harbord or Peters tests are the standard alternatives there; if you only ran Egger, say so as a limit.
• Name the result "small study effects" in the report. Funnel asymmetry also comes from heterogeneity, chance, poorer methods in small studies, and true differences between small and large studies. Publication bias is one explanation among several, and claiming it from a plot alone overstates the evidence.
• Trim and fill, if used, is a sensitivity check that shows how far the estimate could move, presented next to the unadjusted estimate. Presenting the filled estimate as the corrected truth is wrong.

6. Run the sensitivity set

• Leave-one-out: the calculator prints the pooled estimate with each study removed in turn. Flag any study whose removal flips the direction, the significance claim, or the clinical reading, and name it in the report.
• Alternative tau2 estimator and, where the protocol allowed it, the other model.
• Excluding high risk of bias studies, if a risk of bias assessment exists.
• If a sensitivity run changes the conclusion, the conclusion in the write-up is the weaker one, with the reason stated.

7. Write it up to PRISMA 2020

• Report against the PRISMA 2020 checklist (27 items): eligibility, sources, full search strategies, selection and data collection process, risk of bias methods, effect measures, synthesis methods, reporting bias assessment, certainty assessment, then study counts in a flow, per-study results, pooled results with heterogeneity measures, sensitivity results, and certainty per outcome.
• Give registration name and number, where the protocol sits, and every amendment.
• State data and code availability. The study level CSV and the calculator command line are the reproducibility core; publish both when you can.

Output Contract

Deliver all of these:

• The filled protocol, with registration or timestamp and an amendments log.
• The study level data file (CSV in a calculator input shape) with one row per study and the source of each number.
• The calculator report: per-study effects and weights, fixed and random pooled estimates, Hartung-Knapp interval, prediction interval when tau2 is above zero, Q, I2, tau2 and tau, the small study effects section with its limits, and the leave-one-out table.
• A forest plot and a funnel plot (SVG from the calculator is fine).
• A results write-up that follows the reading order: pooled average, interval, prediction interval, heterogeneity in plain words, sensitivity findings, certainty, and a limits paragraph that names the real weaknesses (study count, size spread, bias risk, unassessed small study effects, protocol deviations).
• One headline sentence a reader can quote without lying, for example: "Across 8 studies the average effect was 0.40 (Hartung-Knapp 95% CI 0.11 to 0.69), and the prediction interval (-0.31 to 1.10) says a new setting may see no effect."

Operating Rules

• Protocol before results. Any decision made after seeing pooled output gets labelled as exploratory in the report.
• Never switch models, measures, or estimators because one gives a smaller p value. Run alternatives as sensitivity analyses and report them all.
• Never report I2 without tau, a confidence interval without saying which method built it, or a pooled estimate without k.
• Never run an asymmetry test below 10 studies and present it as a finding. Say the assessment was not reliable instead.
• Never average away opposite direction effects into a near zero pooled number without showing the studies separately.
• Label synthetic or teaching data as synthetic. The worked example in references/worked-example.md uses invented studies; its job is to teach the reading, and it supports no claim about any real intervention.
• Back every method claim in a write-up with a checkable source: Cochrane Handbook for heterogeneity reading, PRISMA 2020 for reporting, the estimator and interval papers for DL, REML, and Hartung-Knapp. Sources used for this skill are listed in the production scratch file and summarised in references/worked-example.md.
• If the data cannot support pooling (overlapping samples counted twice, mixed outcome definitions with no common scale, effects that cannot be placed on one scale), stop, present the studies side by side, and say why pooling was refused.