study_replication_design v1.0.0

name: "study_replication_design" description: "Design a rigorous replication of a published study: extract the target effect, choose the replication type, keep design fidelity, power it honestly, pre-register before data collection, and report failed replications cleanly. Use when asked to replicate a published paper, plan a replication study, pre-register a replication, run a design-fidelity check, or write up a replication that did not work." metadata: { "includeInPrompt": true }

Study Replication Design

Purpose

Turn a published claim into a replication plan that is fair to the original study, honest about power, locked in before data collection, and reported cleanly whether it succeeds or fails. Most failed replications are ambiguous because the plan was sloppy, not because the science was. This skill prevents that.

Workflow

1. Extract the target effect

Read the original paper and fill the target-effect table (see references/worked-replication-plan.md for the template). Record, from the paper itself, never from memory:

• The exact claim being replicated, in one sentence, with the direction of the effect.
• Effect size with its confidence interval, the test used, df, exact p-value.
• Sample size per condition/cell, sampling method, population, exclusions reported.
• Materials, stimuli, timing, instructions, and measurement details for the key DV.
• Anything auxiliary that could matter (time of day, lab vs online, compensation, software).

If the effect size is not reported, compute it from the reported statistics and mark it as derived. If it cannot be derived, say so. Do not guess.

2. Choose the replication type

• Direct replication: same materials, same procedure, same measures, new sample. Default choice. This is the fairest test of the claim.
• Close replication: unavoidable changes (language, population, lab context) documented up front. List every change and why each is unlikely to change the effect.
• Conceptual replication: tests the same hypothesis with different operationalizations. Change only one or two things at a time, or a null result tells you nothing.

Record the choice and its justification in the plan. When in doubt, go more direct.

3. Build the fidelity plan

• Get the original materials. Contact the authors for stimuli, scripts, and measures. If they do not respond, rebuild from the method section and note every reconstruction decision.
• Run a small pilot to check the procedure works end to end (timing, comprehension, no ceiling or floor effects on the DV).
• Include positive controls when possible: a manipulation check or a known effect in the same session, so a null result cannot be blamed on a broken procedure.
• Match exclusion criteria to the original or justify the change before collecting data.

4. Run the power worksheet honestly

Use bin/rep_power.py. The rules:

• Never power a replication on the published effect size. Published estimates from small studies are inflated by selection.
• Compute d33%: the effect size the original study had 33% power to detect (Simonsohn, 2015). A replication with 2.5x the original per-cell sample size has roughly 80% power to reject the "detectable effect" hypothesis. Treat 2.5x as the floor, not the target.
• Power the primary test to 90% or higher against d33% and against a discounted estimate (half the published d is a reasonable discount).
• Run the design analysis (Gelman & Carlin, 2014): for each assumed true effect, report power plus the Type S (wrong-sign) error rate and the Type M (exaggeration) ratio. If power is under 0.5 at your best-guess true effect, the design is too noisy to trust a significant result. Redesign, do not proceed.
• Pre-specify a smallest effect size of interest (SESOI) and plan a TOST equivalence test (Lakens, Scheel & Isager, 2018), so a null result can mean "too small to matter" instead of "inconclusive."

5. Pre-register before touching data

Draft the pre-registration with references/preregistration-checklist.md. Lock all of it:

• Hypothesis, primary DV and measurement, conditions, exact primary analysis.
• Sample size, stopping rule, no peeking.
• Exclusion and outlier rules decided now.
• What counts as a successful replication (significance in the same direction, effect-size CI relative to the original, meta-analytic combination), and what a failed replication means.
• The TOST bounds from step 4.

Register on OSF Registries, AsPredicted, or via a Registered Report Stage 1 protocol. Record the registration link and timestamp in the plan. Treat the pre-registration as a contract; any deviation gets reported and explained, not hidden.

6. Execute with fidelity

• Follow the pre-registered protocol. Log every deviation with a timestamp and reason, even trivial ones.
• Do not peek at the primary analysis before the stopping rule is met.
• Keep raw data, materials, and analysis code versioned so an independent researcher can rerun everything.

7. Report, especially when it fails

Follow references/reporting-failed-replications.md:

• Report the pre-registered analysis first, then exploratory analyses clearly labeled as exploratory.
• Report effect sizes with confidence intervals, not just p-values. A failed replication with a tight CI around zero is informative; a failed replication with a wide CI is inconclusive. Say which one you have.
• Report the TOST result against your pre-registered SESOI bounds.
• Report the meta-analytic combination of original and replication.
• Use the JARS-Quant replication studies checklist (Appelbaum et al., 2018) for the writeup. For trials, CONSORT 2010 covers reporting (not design).
• A single failed replication is one data point. Do not write that the effect is "debunked" or "not real" unless the evidence actually supports that. Frame the result as evidence about the effect size under your design, and say what would change your mind.

Output Contract

A completed replication plan delivers:

1. replication-plan.md: target-effect table, replication type and justification, fidelity plan, power worksheet output, pre-registration draft link, materials checklist.
2. power-worksheet.txt: the output of bin/rep_power.py for the study, kept with the plan.
3. reporting-checklist.md: the failed-replication reporting checklist, filled in at writeup time.

Operating Rules

1. Verify every factual claim about the original study against the paper. No invented numbers, dates, or quotes. Unverifiable = cut it or mark it as a guess.
2. The published effect size is an upper bound on what you plan for, never the planning value.
3. The pre-registration is written before data collection. Anything decided after seeing data is exploratory and labeled as such.
4. One failed replication never licenses "debunked." Report the evidence, bound the effect, and name what would change the conclusion.
5. If any step of this workflow cannot be done honestly (no materials available, cannot afford the required sample, original statistics unrecoverable), say so in the plan instead of fudging it. A replication you cannot power is a replication you should not run.