program_outcomes_measurement v1.0.0

---

name: program_outcomes_measurement description: Measure whether a program actually changed things for the people it serves. Use when picking outcomes and indicators, building a logic model, setting a baseline, planning a follow-up, writing an evaluation plan, or reporting results to a funder, board, or team without overstating what the data shows. Covers the outcome chain, indicator registers, claim levels from change description to attribution, missing data and denominators, and honest reporting language.

---

Program Outcomes Measurement

Purpose

• Help an agent design, run, and report a program evaluation that shows change in participants, not just effort spent.

• Separate outputs from outcomes at every step. Outputs count what the program delivered. Outcomes are changes in participants in knowledge, skills, behaviour, status, or level of functioning, following the definitions in the W.K. Kellogg Foundation Logic Model Development Guide.

• Keep every claim at the level the design can support, so a funder or board gets numbers it can trust and a team gets findings it can act on.

• Ship working material, not advice alone: a logic model, an indicator register, an evaluation plan, and a results report built from the templates in references/templates.md, with the arithmetic done by scripts/outcome_calc.py.

Workflow

1. Start from the decision

• Name the decision the evaluation feeds, who makes it, and the date it gets made. An evaluation with no decision attached collects data nobody uses.

• Name the primary user of the findings first, then other audiences. This follows step 1 of the CDC Framework for Program Evaluation in Public Health: engage stakeholders.

• Fix three to five evaluation questions. Include at least one process question about whether delivery happened as designed and who was reached, plus the outcome questions.

2. Describe the program as an outcome chain

• Build a one page logic model with the template in references/templates.md: need, inputs, activities, outputs, then short-term, intermediate, and long-term outcomes, plus context and assumptions. This is step 2 of the CDC framework.

• Work the chain both ways. Start from activities and ask "so what" to reach outcomes. Start from the long-term outcome and ask "how" to check the activities could plausibly produce it.

• Write out the assumptions between each link. An assumption the evaluation never tests is a place the chain can break silently.

• Sequence outcomes by time. Short-term outcomes come first and lead to intermediate ones, which lead to long-term ones. The Kellogg guide gives one timing illustration: short-term outcomes attainable within 1 to 3 years, longer-term within 4 to 6 years, and impact within about 7 to 10 years. Treat that as that guide's framing for grant programs, and set timings that fit the actual program.

3. Pick indicators that prove the outcome

• Give each outcome one or two indicators. An indicator is a specific, observable, measurable marker of the outcome. The CDC framework warns that defining too many indicators detracts from the evaluation's goals, and that an indicator alone is no substitute for a justified conclusion, because conditions outside the program can move it.

• Fill the indicator register in references/templates.md for every indicator: exact numerator and denominator, instrument, timing, owner, target, disaggregation, and known weakness.

• Prefer a marker close to the outcome itself. For a skills outcome, a scored performance marked by someone who did not teach the participant beats a satisfaction rating. Self-report is acceptable when nothing closer is feasible, and the report must say it is self-report.

• Use a validated instrument where one exists for the construct and population. If the team builds its own items, pilot them, check that two markers of the same outcome agree, and record the weakness in the register.

• Add at least one qualitative source for each main outcome, such as interviews or open text, to explain why the numbers moved and to catch changes the indicators miss, including harm.

• Set targets before delivery starts, with the reason for the level written down. A target chosen after results arrive proves nothing.

4. Fix the design and the claim level before collecting data

• Choose the claim level now, in writing, from this ladder:
• Change description: baseline and follow-up on participants, no comparison. Supports wording about what changed, for whom, and by how much.
• Contribution: change description plus a tested theory of change, rival explanations checked, and evidence from more than one source. Supports wording that the program plausibly contributed to the change.
• Attribution: a credible counterfactual, such as a randomised or well matched comparison group, or a phased rollout that creates one. Supports wording that the program caused an estimated amount of change.

• Treat a pre and post design with no comparison as the default for a small program, and label it honestly. It cannot rule out outside events, maturation, or who chose to enrol.

• Plan the follow-up point from the outcome, not from the funding calendar. An outcome defined as lasting change needs a measurement after the program ends, at a gap long enough to test whether the change held.

5. Measure the baseline, then protect the follow-up

• Measure the baseline before delivery starts, or at intake before any teaching, treatment, or service that could shift it. The baseline is the measured condition at the start, and change is measured against it. Without it, later figures show levels, not change.

• Use the same instrument, same scoring rules, and same definitions at baseline and follow-up. If an instrument must change mid-stream, measure a bridge sample both ways or report the two periods separately.

• Define the eligible group on day one and keep the denominator fixed. Every rate in the report carries its numerator and denominator.

• Chase missing follow-up data with a set protocol: repeat contacts, more than one contact mode, and a short version of the instrument for people who will not do the full one.

• Compare respondents with non-respondents on what is already known, such as baseline score, attendance, and subgroup. State plainly which way the missing data probably pushes the results.

6. Analyse and compute honestly

• Run the arithmetic with scripts/outcome_calc.py. It prints absolute change, relative change, progress to target, response rate, and the missing share, and it works for lower-is-better outcomes with the --lower-is-better flag:

python3 scripts/outcome_calc.py --baseline 42 --followup 57 --target 70 --eligible 60 --responded 49 --unit percent

• Report absolute change first. Add relative change only when the baseline is not zero and not tiny, because a rise from 2 percent to 4 percent is 2 points, and calling it a doubling misleads.

• Report paired figures as paired. A mean gain among people with both measurements describes that group, and the report names the group size beside the gain.

• Break results out by the subgroups named in the plan, and only those. Post hoc subgroup hunting finds noise. Where a subgroup is small, give the count and hold back strong wording.

• Weigh practical size alongside any statistical test. A change can be statistically detectable and too small to matter for the decision, or large and worth acting on with wide uncertainty. Report the size, the spread or interval where the data supports one, and the sample size.

• Check the process data before explaining the outcome data. Low reach or weak fidelity explains a flat outcome better than a failed idea does, and the two call for different fixes.

7. Justify conclusions and report for use

• Test rival explanations before writing findings: outside events in the period, other services participants used, seasonal patterns, changes in who enrolled, and measurement drift. This is contribution analysis in practice: test the chain, test the alternatives, and combine sources before claiming a role for the program.

• Write each finding at the claim level fixed in step 4. Use the wording test in references/worked-example.md, which shows supported and unsupported sentences side by side for the same numbers.

• Put limitations in the main body of the report: who is missing, what was not measured, which outcomes have no follow-up yet, and what else could explain the change.

• Use the report skeleton in references/templates.md. Keep one table per outcome with baseline, follow-up, change, target, and the counts behind every rate.

• Deliver findings before the decision date, in a format the primary user can act on, and book the meeting where findings get used. This is step 6 of the CDC framework: ensure use and share lessons learned.

Output Contract

• Deliver these, in this order:
• A one page logic model using the template, with assumptions and context filled in.
• An indicator register with no more than two indicators per outcome, each with definition, instrument, timing, owner, target, and known weakness.
• A short evaluation plan naming the decision, users, questions, design, claim level, data handling, analysis, reporting date, and the use meeting.
• Baseline and follow-up figures computed with scripts/outcome_calc.py or checked against it, with numerators and denominators shown.
• A results report using the skeleton, with limitations in the main body and every finding worded at the planned claim level.

• State what was verified and what remains unmeasured. If a follow-up point has not happened yet, the report says the outcome is not yet known, not that it is on track.

Operating Rules

• Never present an output count as an outcome. Sessions held and people reached describe delivery. They belong in the process section.

• Never use causal wording ("caused", "led to", "raised by") unless the design includes a credible counterfactual. Default to "changed", "was associated with", or "the evidence is consistent with a contribution from the program" as the design allows.

• Never quote a percentage without its numerator and denominator in the same table or sentence.

• Never drop non-respondents or dropouts from a rate silently. Report results on the enrolled or eligible base as well as on respondents where the two differ.

• Never move a target, swap an indicator, or change a definition after seeing results without disclosing the change and the reason in the report.

• Never report a relative change on a zero or near zero baseline. Give the absolute change.

• Protect participants. Collect only the data the questions need, get consent that matches the actual use, store individual records with named access, and report small subgroups in a way that cannot identify a person.

• Check the plan against the four CDC standard groups before data collection: utility (will the primary user get usable findings in time), feasibility (can this team collect this well with its time, resources, and expertise), propriety (are rights, welfare, and affected groups protected), and accuracy (will the design support the claim level chosen).

• Keep the evaluation sized to the program. A small program with a two person team gets a tight register, a clean baseline, and one follow-up done properly, rather than a wide register done badly.

Sources

• CDC, Framework for Program Evaluation in Public Health, MMWR 1999: https://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm
• CDC, Program Evaluation Framework Action Guide, step 2, Describe the Program: https://www.cdc.gov/evaluation/php/evaluation-framework-action-guide/step-2-describe-the-program.html
• CDC, Developing a Logic Model: https://www.cdc.gov/tb-programs/php/evaluation/logic-model.html
• W.K. Kellogg Foundation, Logic Model Development Guide, 2004: https://dl.icdst.org/pdfs/files/0d84f6f39fca28a8a08f2870a402a017.pdf
• Contribution analysis chapter, Policy Evaluation: Methods and Approaches: https://scienceetbiencommun.pressbooks.pub/pubpolevaluation/chapter/contribution-analysis/
• Response rates and responsiveness for surveys, standards review: https://pmc.ncbi.nlm.nih.gov/articles/PMC2384218/