causal_root_cause_analysis v1.0.0

name: "causal_root_cause_analysis" description: "Test what actually caused a change in observational data: state competing explanations first, then test each with difference in differences, regression discontinuity, instrumental variables, or matching, plus negative controls, grade how strong the evidence behind each test is, and report what the evidence rules out. Triggers when a metric moved after a policy, programme, price, or product change and someone asks whether the change caused it, when a before and after or treated versus not comparison is being read as proof, or when an observational study needs a causal audit before a funding, rollout, or rollback decision." metadata: { "includeInPrompt": true }

Causal Root Cause Analysis

Purpose

Answer one kind of question well: did this change cause that movement, or would it have happened anyway? The skill runs a causal audit on observational data. It names the rival explanations first, picks a design that fits how treatment was actually assigned, runs the diagnostics that design lives or dies on, checks negative controls, grades each test, and reports what the evidence rules out, what it leaves open, and what follows for the decision.

A correlation with controls added is not a design. Each of the four designs here buys its causal claim with a different assumption, and the report has to name that assumption in plain words, show the check that tests what can be tested, and mark what rests on argument alone.

The full worked case is in references/worked-case.md. It follows one funding question through all four designs on seeded synthetic data made by scripts/causal_audit.py, including a matching analysis that overstates the effect and gets caught by its own negative control. The design cards and grading rubric are in references/design-cards.md.

Workflow

1. Write the decision and the estimand before touching data. State what will be done differently for each possible answer, whose effect is needed (everyone, the treated, compliers, or units near a cutoff), and over what period. If the available design can only estimate a narrower effect than the decision needs, say so at the start, not in a footnote at the end.

1. List competing explanations before running anything. Write at least four: the treatment worked, a shared trend or season did it, the treated units were different to start with, something else changed at the same time, plus any setting specific rival such as bounce back after a bad period, a measurement change, or spillover between groups. For each rival, write the pattern in the data that would support it. A test earns its place by ruling a named rival out.

1. Reconstruct how treatment was assigned, from facts, not from the dataset alone. Who chose, what rule or score was used, what else changed on the same date, whether units could see treatment coming and react early, and whether treated units could affect comparison units. Assignment facts choose the design. If nobody can say how treatment came about, no design in this skill can rescue the question, and the report says that.

1. Sketch the causal structure in words or a diagram. Name the confounders a design must handle, the mediators that must not be adjusted away when the total effect is the target, and any collider that adjustment would open. For matching, this step decides which traits belong in the match: pre treatment causes of both treatment and outcome, never consequences of treatment.

1. Pick the design that matches the assignment story. Use the selection rules in references/design-cards.md. A rollout with a comparison group points to difference in differences. A score and a cutoff point to regression discontinuity. A randomised encouragement or another plausibly external push points to instrumental variables. Rich measured traits with good overlap point to matching, which starts with the weakest standing and needs the most support from other checks. When two designs are available, run both. Agreement across designs with different weak points is worth more than a second decimal place from one design.

1. Run the diagnostics for the chosen design before quoting its estimate. Difference in differences: pre period trend plot, event study read, staggered adoption check. Regression discontinuity: outcome plot at the cutoff, density check for sorting, trait balance at the cutoff, at least three bandwidths. Instrumental variables: first stage size reported plainly, placebo checks on groups the instrument cannot move. Matching: standardised mean differences before and after for every trait, overlap counts, and how many treated units were dropped. The commands in scripts/causal_audit.py produce the core numbers for all four designs from plain CSV files.

1. Run at least one negative control for every design. Swap in an outcome the treatment cannot cause but that shares the feared confounding path, or rerun with a placebo treatment date in the pre period. Pre write why the treatment cannot affect the control, including indirect routes. A non null control result is a finding about the design, not an embarrassment to hide. Report it next to the main estimate, at the same size.

1. Run sensitivity checks that map to the actual threats. Vary the bandwidth, the caliper, the comparison group, or the period window, one at a time, and record how far the estimate moves. For matching and other selection on observables designs, state how strong an unmeasured factor would need to be to explain the result away. Skip checks that answer a threat nobody raised for this setting, and say which real threat has no available check.

1. Grade each test separately using the rubric in references/design-cards.md. Grades run A to D and turn on four things: whether the assignment story supports the identifying claim, whether the diagnostics passed, whether the negative control came back null, and whether the estimand matches the decision population. Never average grades across designs and never let a precise confidence interval raise a grade. Precision is not identification.

1. Reconcile disagreement by evidence, not by averaging. When designs disagree, check each one for a kill condition or a failed control, set aside any design that has one, and explain the failure in the report, as the worked case does with matching. What remains is a range across surviving designs, with each estimand named, because exact agreement between different estimands would be a coincidence.

1. Close with the ruled out list and the decision. State which competing explanations the evidence rules out and for which population, which explanations stay live and what would settle them, and the recommendation that follows at the current grades. If no design reaches grade B, the honest output is an adjusted association, a clear statement that the causal question is open, and the data or rollout change that would close it, such as randomising encouragement, staggering rollout, or recording the assignment score.

Output Contract

Deliver a causal audit report with these parts, in this order:

• Question and decision: the change, the outcome, the population, the period, and what action each answer would trigger.
• Competing explanations: the list from step 2, each with the data pattern that would support it.
• Assignment account: how treatment was actually assigned, and which facts that account rests on.
• Test table: one row per design run, with the estimand in plain words, the estimate with its units and uncertainty, the key diagnostics with their results, the negative control result, and the grade.
• Ruled out and still live: which explanations the evidence excludes, for whom, and which remain open with the reason.
• Decision line: the recommendation, the grade of the strongest surviving evidence behind it, and what new data would raise that grade.
• Reproducibility note: the data files, the script commands run, and any synthetic or illustrative numbers clearly labelled as such.

Operating Rules

• Never present a before and after change, or a treated versus untreated gap, as a causal effect. Those are the patterns the audit exists to test.
• Never invent an estimate, a diagnostic result, a sample size, or an assignment fact. Every number in the report comes from the data in hand, a named run of the script, or a labelled synthetic example.
• State the identifying assumption of every design in plain words, and mark which parts of it the data can check and which parts rest on argument. Exclusion restrictions and parallel trends after treatment are never proven by a test, and the report must not imply they are.
• Report negative control and placebo results at the same prominence as main results. Burying a failed control converts the report from an audit into advocacy.
• Name the estimand every time an estimate appears. A cutoff effect, a complier effect, and an effect on the treated are different answers, and quoting one as if it were the effect for everyone is a factual error, not a simplification.
• With staggered adoption and effects that may vary across cohorts or time, do not report a plain two way fixed effects coefficient as the difference in differences answer. Use an estimator built for staggered designs or show the decomposition.
• Do not use balance test p values as the matching balance standard. Use standardised mean differences and distributional checks, and report dropped units.
• Do not adjust for post treatment variables when the total effect is the target, and do not match or condition on a collider. If a requested adjustment would do either, flag it and explain the cost before running it.
• If the data cannot support any design at grade B or better, say the causal question is unanswered by this data. Give the association honestly labelled, and specify the smallest change to data collection or rollout that would make the question answerable.
• Keep the synthetic worked case labelled as synthetic wherever its numbers are reused. Its true effects are known because they were built in, which is exactly what real data never offers.