survey_data_analysis v1.0.0

name: "survey_data_analysis" description: "Analyze survey responses after collection: clean the data, apply post-stratification weights, build cross-tabs, and run basic significance tests with honest interpretation of non-significant results. Use when you have survey responses in a file and need trustworthy numbers from them." metadata: { "includeInPrompt": true }

Purpose

Turn raw survey responses into numbers you can defend. Cleaning, weighting, cross-tabs, and basic significance testing, with plain rules for saying "this is not significant" instead of dressing noise up as a finding.

Not this skill: survey-design-basics is "Design quantitative surveys that give usable answers: write unbiased questions, pick the right question type, size the sample, compute response rates, and check for bias. Use when building a poll, questionnaire, or feedback survey whose results will be counted." That skill owns everything up to the moment responses arrive. This one starts there.

Workflow

1. Shape the data. The script below expects a CSV named responses.csv with one row per response and at least these columns: respondent_id, age_group, q_sat, minutes. Adapt column names in the script if yours differ. Keep the raw file untouched; the script never edits it.

1. Save the reference script and run it. Copy the block below into analyze.py, put it next to responses.csv, then run python3 analyze.py. It uses only the standard library (csv, math), so it runs anywhere.

python
    import csv, math

    # ---------- config ----------
    POP_SHARES = {"18-34": 0.42, "35-54": 0.38, "55+": 0.20}  # known population age mix
    SATISFIED = {"Very satisfied", "Satisfied"}
    SPEEDER_CUTOFF_MIN = 1.0
    Z95 = 1.96

    # ---------- load ----------
    with open("responses.csv") as f:
        raw = list(csv.DictReader(f))

    # ---------- 1. cleaning ----------
    seen, deduped = set(), []
    dups = 0
    for r in raw:
        if r["respondent_id"] in seen:
            dups += 1
            continue
        seen.add(r["respondent_id"])
        deduped.append(r)

    clean, speeders, missing = [], 0, 0
    for r in deduped:
        if float(r["minutes"]) < SPEEDER_CUTOFF_MIN:
            speeders += 1
            continue
        if not r["q_sat"].strip() or not r["age_group"].strip():
            missing += 1
            continue
        clean.append(r)
    n_clean = len(clean)

    # ---------- 2. post-stratification weights ----------
    stratum_n = {}
    for r in clean:
        stratum_n[r["age_group"]] = stratum_n.get(r["age_group"], 0) + 1
    weights = {s: POP_SHARES[s] / (stratum_n[s] / n_clean) for s in POP_SHARES}

    def kish_n_eff(ws):
        return sum(ws) ** 2 / sum(w * w for w in ws)

    for r in clean:
        r["w"] = weights[r["age_group"]]
        r["sat"] = 1 if r["q_sat"] in SATISFIED else 0

    # ---------- 3. overall weighted estimate + margin of error ----------
    sw = sum(r["w"] for r in clean)
    p_overall = sum(r["w"] * r["sat"] for r in clean) / sw
    n_eff_all = kish_n_eff([r["w"] for r in clean])
    moe = Z95 * math.sqrt(p_overall * (1 - p_overall) / n_eff_all)

    # ---------- 4. cross-tab: weighted satisfied % by age group ----------
    crosstab = {}
    for s in POP_SHARES:
        g = [r for r in clean if r["age_group"] == s]
        swg = sum(r["w"] for r in g)
        p = sum(r["w"] * r["sat"] for r in g) / swg
        crosstab[s] = (len(g), p, kish_n_eff([r["w"] for r in g]))

    # ---------- 5. two-proportion z-test: 18-34 vs 55+ ----------
    def norm_cdf(x):
        return 0.5 * (1 + math.erf(x / math.sqrt(2)))

    n1, p1, ne1 = crosstab["18-34"]
    n2, p2, ne2 = crosstab["55+"]
    x1, x2 = p1 * ne1, p2 * ne2
    p_pool = (x1 + x2) / (ne1 + ne2)
    se = math.sqrt(p_pool * (1 - p_pool) * (1 / ne1 + 1 / ne2))
    z = (p1 - p2) / se
    p_value = 2 * (1 - norm_cdf(abs(z)))

    # ---------- report ----------
    print(f"raw rows: {len(raw)}")
    print(f"duplicates removed: {dups}")
    print(f"speeders dropped (<{SPEEDER_CUTOFF_MIN} min): {speeders}")
    print(f"rows dropped for missing answers: {missing}")
    print(f"clean n: {n_clean}")
    print()
    print("post-stratification weights (stratum: weight):")
    for s in POP_SHARES:
        print(f"  {s}: n={stratum_n[s]} sample_share={stratum_n[s]/n_clean:.3f} "
              f"pop_share={POP_SHARES[s]:.2f} weight={weights[s]:.3f}")
    print()
    print(f"overall satisfied: {p_overall*100:.1f}% +/- {moe*100:.1f}pp (95% MoE, effective n={n_eff_all:.0f})")
    print()
    print("cross-tab: satisfied % by age group (weighted):")
    for s in POP_SHARES:
        n, p, ne = crosstab[s]
        print(f"  {s}: n={n} -> {p*100:.1f}% (effective n={ne:.0f})")
    print()
    print(f"two-proportion z-test, 18-34 vs 55+: z={z:.3f}, p={p_value:.3f}")
    print("conclusion: " + ("difference IS statistically significant at 95%."
          if p_value < 0.05 else
          "this difference is NOT statistically significant at 95%. "
          "Write that plainly: the data does not support a real gap between these groups."))
    

1. Read the cleaning report at the top of the output. It tells you how many rows were raw, how many duplicates, speeders, and incomplete rows were dropped, and the clean n. Quote these counts in your write-up. A worked run on a 193-row fixture: 2 duplicates removed, 5 speeders dropped, 12 rows dropped for missing answers, clean n of 174.

1. Read the weights and cross-tab. Each weight equals the population share of the stratum divided by its share of the clean sample (for example, the 55+ group came out at 0.20 / 0.374 = 0.535, while the under-represented 18-34 group got 1.555). The script uses Kish effective sample size, (sum of weights)^2 / (sum of squared weights), wherever a plain n would overstate precision. The worked run gave 60.7% satisfied overall with a 95% margin of error of 7.8 points (effective n of 149), and weighted cross-tab values of 55.3% for 18-34 (n=47), 72.6% for 35-54 (n=62), and 49.2% for 55+ (n=65).

1. Test the comparison and write the honest sentence. The script runs a two-proportion z-test on the effective counts, with the p-value from the normal CDF built on math.erf. If p is 0.05 or higher, the comparison is not significant and you say so in plain words. Worked example, 18-34 vs 55+: z = 0.636, p = 0.525. The sentence to write: "This difference is not statistically significant at 95%. The data does not support a real gap between these groups." Do not follow it with "but the trend suggests" or any softened version of the same claim.

1. Sanity-check anything that will be quoted. Recompute at least one weight and one percentage by hand from the printed counts before the numbers leave your hands.

Output Contract

Every analysis you hand over includes: the cleaning counts (raw, dropped, clean n), the weighting scheme (strata, population shares used, resulting weights), each reported estimate with its margin of error and effective n, the cross-tab behind any group comparison, and the test statistic plus p-value for every claimed difference. Non-significant results are stated plainly, in the same prominence as significant ones. Nothing is reported from the raw unweighted counts alone when weights were applied.

Operating Rules

• Report the effective sample size, not the raw row count, next to every weighted number.
• A p-value of 0.05 or higher ends the claim. No "directional", "suggestive", or "approaching significance" language.
• Never compare groups the cross-tab was not built for, and never slice the data further after seeing a non-significant result to hunt for a slice that passes.
• If a stratum has fewer than 30 clean responses, say the estimate for that group is shaky and widen the caution, even if the math runs.
• Keep formulas visible: margin of error is 1.96 sqrt(p (1 - p) / n_effective), and any reader should be able to redo it from your reported numbers.