survey_design_basics v1.0.0

name: "survey_design_basics" description: "Design quantitative surveys that give usable answers: write unbiased questions, pick the right question type, size the sample, compute response rates, and check for bias. Use when building a poll, questionnaire, or feedback survey whose results will be counted." metadata: { "includeInPrompt": true }

Survey Design Basics

Purpose

Turn a vague data need into a survey whose results you can trust. This skill owns QUANTITATIVE surveys: structured questions with countable answers.

Not this skill: use research-methods to evaluate evidence quality and cite sources, and customer-interviews to run conversational discovery about real problems.

Workflow

1. Define the single decision the survey must inform. Write it in one sentence. If you cannot, stop and clarify before writing questions.
2. Define the population (who you ask) before anything else. A perfect questionnaire asked to the wrong group is worse than a rough one asked to the right group.
3. Pick question types for each data need (see below). Write neutral questions. Fix the three wording traps with the before/after patterns.
4. Size the sample with the calculator, then inflate the send list for the expected response rate.
5. Pre-test on 5 to 10 real respondents. Rewrite anything they misread.
6. Run bias checks after collection: nonresponse, selection, and order effects.

Question wording: the three traps

Leading questions push an answer.

• Bad: "How much do you love our new checkout?"
• Good: "How would you rate our new checkout?"

Double-barreled questions ask two things at once. Answers are unreadable.

• Bad: "Was the staff friendly and the food good?"
• Good: split into "How friendly was the staff?" and "How good was the food?"

Ambiguous wording lets respondents answer different questions. Words like "often", "recently", "sometimes" mean different things to different people.

• Bad: "Do you visit us often?"
• Good: "How many times did you visit in the last 30 days?"

Question types and when each fits

• Multiple choice (single select): when the answer set is small and known. Keep options mutually exclusive and exhaustive (always add "Other" or "None of these").
• Likert scale: when you need intensity on one dimension (agreement, satisfaction). Use 5 points with labeled endpoints. Keep all items in the same direction or recode before analysis.
• Open text: when you do not know the range of answers. Cap at 2 per survey; they are expensive to code and most respondents skip them.
• Ranking: when relative priority is the point ("rank these 4 features"). Keep lists to 5 or fewer or the results turn into noise.

Sampling basics

Who you ask matters more than how many. A sample of 385 from the right population beats 10,000 from the wrong one. List the exact group the decision applies to, then recruit only from it. Convenience samples (your followers, your mailing list) are fine if you name the limitation honestly in the results.

Sample size and response-rate math

For a proportion at 95% confidence with maximum uncertainty (p = 0.5):

n = Z^2 * p(1 - p) / E^2, with Z = 1.96.

Computed on this machine:

• E = 0.05 (plus or minus 5 points): n = 385
• E = 0.03 (plus or minus 3 points): n = 1068

You need the FULL n back as responses, so inflate the send list:

send = n / expected response rate, rounded up.

Worked example: you need 385 responses and expect a 32% response rate. 385 / 0.32 = 1203.1, so send 1204 invites. If only 384 come back out of 1200 sent, your response rate is 384 / 1200 = 32.0%.

Calculator (save as calc.py, run with python3 calc.py):

python
import math

Z = 1.96  # 95% confidence

def sample_size(margin, p=0.5):
    return math.ceil(Z**2 * p * (1 - p) / margin**2)

def send_list(need, rate):
    return math.ceil(need / rate)

def margin_of_error(n, p=0.5):
    return Z * math.sqrt(p * (1 - p) / n)

print("n for +/-5%:", sample_size(0.05))
print("n for +/-3%:", sample_size(0.03))
print("invites for 385 at 32% rate:", send_list(385, 0.32))

Output on this machine: n for +/-5%: 385. n for +/-3%: 1068. Invites: 1204.

Bias checks

• Nonresponse bias: the people who answer differ from those who do not. Compare early vs late respondents; if they disagree, your tail is different from your head. Example computed above: if 60% of customers bought and 90% of buyers are satisfied, but only 40% of non-buyers are, true satisfaction is 70%, not the 90% the replies show.
• Selection bias: your recruiting channel filtered the population. A survey pushed through your app only measures app users, not all customers. State the reachable population, not the intended one.
• Question-order effects: earlier questions prime later answers. Never ask a satisfaction question right after a complaint prompt unless that is the point. Rotate item order in a pilot and compare.

Output Contract

Deliver: the decision sentence, the population definition, the question list (each labeled with its type), the computed n and send count with the response-rate assumption, the pre-test results, and the three bias checks with findings.

Operating Rules

1. One decision per survey. Two decisions need two surveys or one split by a clear section break.
2. Every question must survive this test: "What will I do differently depending on the answer?" If nothing, cut it.
3. Keep surveys under 5 minutes. Each extra question lowers the completion rate.
4. Never report a percentage without its base: "72%" means nothing; "72% of 385 respondents" does.
5. Never round n down. Round sample sizes up.
6. Name the reachable population in every result writeup. A biased sample reported honestly still teaches something; a biased sample reported as "customers" misleads.