usability_testing_basics v1.0.0

name: usability_testing_basics description: "Plan and run usability tests: write a test plan, design goal-based tasks, moderate think-aloud sessions, rate finding severity, and produce a findings report. Use when testing whether people can complete tasks in a product, prototype, or flow." metadata: { "includeInPrompt": true }

Usability Testing Basics

Purpose

Plan and run a usability test, then turn what happened in the sessions into a findings report a team can act on. This skill covers the test itself: the test plan, task design, think-aloud moderation, note-taking, severity rating, and the findings report.

Scope:

• customer-interviews is product discovery interviews; survey-data-analysis is analyzing survey responses; interview-transcript-analysis codes transcripts after sessions. This skill owns planning and running usability tests themselves: test plan, task design, think-aloud moderation, severity rating, and a findings report.
• No usability skill exists in the catalog at all, so this skill is the starting point for task-based testing work.
• This skill covers moderated tests, in person or over a video call, and notes for unmoderated tests where the same task and metric rules apply. It does not cover recruiting services, survey design, or interview technique for discovery research.

Workflow

1. Write the test plan before recruiting anyone. Fill in every field. A field left blank is a decision the moderator will have to invent mid-session.

• Goal: the one decision this test informs, in one sentence. Example: decide whether the new checkout flow is ready to ship.
• Research questions: 2 to 4 questions the sessions must answer. Example: can a first-time user complete checkout without help?
• Product under test: product name, version or build, prototype fidelity if it is a prototype, and which parts work and which parts are stubbed.
• Participants: who qualifies (role, experience level, prior use of the product), who is excluded, target count (see step 2), and the screening questions used to check fit.
• Tasks: the task list from step 3, in the order participants will attempt them, with the success criterion for each task.
• Metrics: which of the success metrics in step 7 will be recorded for each task.
• Session setup: session length (30 to 60 minutes is a common working range), device and screen size, starting state (logged in or out, empty or seeded data), test accounts or test data to use, and what gets reset between sessions.
• Roles: moderator, note-taker, and any observers. One person can hold two roles in a small test; write down who holds which.
• Consent and recording: whether sessions are recorded, what participants are told before starting, and where recordings and notes are stored.
• Pilot: date of the pilot session (step 4) and who reviews the pilot before the full run.
• Schedule and deliverable: session dates, and the date the findings report is due.

1. Choose a participant count and state the reason.

• Participant count is guidance, not law. Pick a count that matches the claim the report will make, and write the reasoning into the plan.
• For finding problems in one design for one audience, small samples find many issues. A common heuristic is about 5 participants per distinct user group: early participants tend to surface the most common, highest-impact problems, and later participants repeat findings more often than they add new ones.
• Treat the 5-user figure as a heuristic with limits. It does not hold when the audience splits into groups with different tasks or experience, when the product has many separate flows, or when a problem is rare but costly. Test each distinct user group separately, and add participants when new sessions are still producing new high-severity findings.
• For quantitative claims, such as a completion rate or an average time on task that will be compared against a target or a prior version, small samples are too noisy. Use a larger sample, state the target and the sample size in the plan before running sessions, and report results with the sample size attached. Do not present a rate from 5 sessions as a stable product metric.
• Stop rule for problem-finding tests: stop when two sessions in a row add no new finding at severity 2 or higher, or when the planned count is reached. Record which rule stopped the test.

1. Write the tasks.

• Write goals, not instructions. A task states what the participant is trying to achieve and gives a realistic reason, without naming the screens, menus, buttons, or steps used to get there.
• Weak: "Click Search, set the price filter to under $50, and press Apply."
• Stronger: "You want to buy a gift for a coworker and spend under $50. Find something that fits."
• Use no leading wording. Do not use the product's own label for a feature in the task if finding that feature is part of what is being tested. Do not hint that a task is easy, hard, new, or recently changed.
• Give each task: an ID, the wording read to the participant, a starting point, a clear success criterion (the end state that counts as complete), and a time or step limit after which the moderator ends the task and marks it failed or incomplete.
• Order tasks from a natural starting task to later ones, and avoid making a later task depend on an earlier task succeeding. Where a dependency is unavoidable, give the participant the needed end state and say so in the notes.
• Keep the count small enough to fit the session with time for follow-up questions. Four to six tasks in a 45-minute session is a workable starting point; cut tasks before cutting the time each task needs.
• Add one warm-up item at the start: a short question about the participant's relevant habits, or a trivial first task. It settles the participant and checks the setup.

1. Pilot the test.

• Run one full session with a stand-in participant who matches the target audience as closely as is practical.
• Check: task wording is understood as written, success criteria can be judged from what is observable, the session fits the scheduled length, the recording works, and the reset steps return the product to the starting state.
• Fix wording and setup problems found in the pilot, record the changes in the plan, and exclude pilot data from the findings unless the pilot ran the final version unchanged.

1. Moderate with the think-aloud protocol.

• Opening script, read or paraphrase it the same way each session:
• "We are testing the product, not you. There are no wrong answers, and you cannot fail this test."
• "Please say out loud what you are looking at, what you are trying to do, and what you expect to happen as you go. If you stop talking, I may remind you to keep talking."
• "I cannot help with the tasks or answer questions about how the product works during the test. If you ask, I will probably say: what do you think it does? I will answer questions at the end."
• "You can stop any task, or the session, at any time."
• Moderator rules during tasks:
• Stay neutral. Use prompts that keep the participant talking without steering: "What are you thinking now?", "What did you expect to happen?", "What are you looking for?"
• Do not help. Do not explain the interface, confirm a choice is right, point at the screen, or react to mistakes. Do not praise a specific action; a flat "thank you, keep going" is enough.
• Do not defend the design. If a participant criticizes the product, acknowledge it and ask what they expected. Record the criticism as data.
• Let silence and struggle run for a short while before prompting. A participant working a problem out is evidence; interrupting it loses the evidence. End a task at its planned limit, note where it ended, and move on without framing it as a failure by the participant.
• If the participant asks whether they did it right, return the question: "What makes you ask?" or "What did you expect?" Save real answers for after the last task.
• Ask follow-up questions after each task, not during: what was hardest, what they expected instead, and how confident they are that they completed it.
• Close the session: ask an overall impression question, answer the participant's saved questions, thank them, and stop the recording. Write a short session summary within a day, while details are still fresh.

1. Take notes on a fixed grid. Use one row per task per participant. Fixed columns keep sessions comparable and make the tally in step 8 possible.

• Columns: participant ID, task ID, outcome (complete, complete with difficulty, incomplete, failed), time on task, error count, assists given (should be zero; record any exception), what the participant expected, what actually happened, a short quote in the participant's words, and observer notes.
• Record behavior first and interpretation second, in separate wording. "Clicked the logo three times looking for home" is behavior. "Navigation model is unclear" is interpretation, and belongs in the findings, tied to rows like that one.
• Note-taker and moderator compare notes after each session and agree on the outcome code for each task while the session is fresh.

1. Record the success metrics per task.

• Task completion: completed tasks divided by attempted tasks, reported per task and per participant, with the sample size. Count "complete with difficulty" separately from clean completion; merging them hides friction.
• Errors: count a defined error event, and define it in the plan. A workable definition: an action that moves the participant away from the success criterion, such as opening the wrong section, submitting wrong data, or using back or undo to recover. Count repeated identical actions once per attempt at that step.
• Time on task: time from the end of reading the task to reaching the success criterion or to the task being ended. Report the range and the middle value across participants rather than only an average, since one long struggle skews an average in a small sample.
• Optional per-task rating: after each task, ask the participant to rate how hard the task felt on a 1 to 7 scale, where 1 is very difficult and 7 is very easy. Report it as the participant's perception, separate from whether they completed the task.
• Report every metric with its sample size and the product version tested.

1. Turn notes into findings and rate severity.

• Group note rows into findings: one finding is one underlying problem, supported by the rows where it appeared. A problem seen in one session is still a finding; record how many participants hit it and say when a count is 1.
• Write each finding as: a title that names the problem, the task or tasks affected, what happened, the evidence (participant count, quotes, metric values), and a recommended fix stated as a direction, not a finished design.
• Rate severity with this scale. This scale is this skill's convention; state it in the report so readers do not mistake the numbers for an industry standard.
• 4: Task blocker. The participant cannot complete the task, or completes it with wrong results they do not notice. Fix before release.
• 3: Major problem. The participant completes the task only with great difficulty, repeated errors, or a workaround. Fix on the current cycle.
• 2: Minor problem. The participant completes the task but hesitates, takes a detour, or is confused and recovers. Fix when the area is next touched.
• 1: Cosmetic or polish issue. Completion is unaffected. Examples: alignment, contrast, wording polish. Fix opportunistically.
• 0: Not a problem. A positive observation, a preference with no completion effect, or a suspected issue the evidence did not support. Kept in the list so it is not re-raised as new.
• Rate by impact on completion and how many participants were affected, not by how easy the fix looks or who reported it. When two raters disagree, keep the higher rating if the evidence shows a participant actually blocked, and record the disagreement in the finding.
• Tally the findings with the script in the Operating Rules section before writing the report, and use its counts in the report summary.

Output Contract

Deliver a findings report with these sections, in this order:

1. Summary: the decision the test informed, product and version tested, number of participants and sessions, dates, and the headline result in 3 to 5 bullet points.
2. Severity tally: total findings and the count at each severity level, produced by severity_tally.py. On the worked fixture in the Operating Rules section the tally is: total 8, severity 4: 2, severity 3: 3, severity 2: 1, severity 1: 1, severity 0: 1.
3. Metrics table: one row per task with attempts, clean completions, completions with difficulty, failures, error count, and time on task (range and middle value). Every value carries the sample size from the plan.
4. Findings: one entry per finding, sorted by severity from high to low, each with ID, title, severity, task, evidence (participants affected, quotes, metric values), and a recommended fix direction.
5. What worked: patterns participants completed cleanly or praised, stated briefly so a redesign does not remove them.
6. Method note: the test plan version, participant count and the reasoning for it, the severity scale used, any deviations from the plan, and sessions excluded and why.
7. Limits: what this test cannot support, such as population-level completion rates from a small sample, audiences or flows not tested, and prototype parts that were stubbed.

A sample finding entry:

• ID: F-01
• Title: Price filter resets after applying it
• Severity: 4
• Task: Find a gift under $50
• Evidence: 2 of 5 participants applied the price filter, saw the full price list return, and concluded no gifts matched. One said: "I set under fifty and it just went back to everything."
• Recommended fix direction: keep applied filters in place after applying, and show the active filter state on the results view.

Operating Rules

1. Test the product, and say so. Every session opens with the framing that the product is under test and the participant cannot fail. If a participant leaves feeling they performed badly, the moderation failed, not the participant.
2. No helping during tasks. Any assist is recorded in the note grid with the reason, and that task's outcome is marked as assisted rather than complete.
3. Numbers carry their sample size. Completion rates, times, and error counts from a small problem-finding sample describe those sessions. Do not restate them as rates for all users.
4. The 5-user figure stays a heuristic in every plan and report. Present it with its limits from Workflow step 2, never as a guaranteed number that finds a fixed share of problems.
5. Quotes are verbatim or marked as cleaned. Keep the participant ID and task ID attached to each quote so it can be traced back to the session notes.
6. Severity ratings use the 0 to 4 scale defined in Workflow step 8, and the report prints that definition or links the section of the plan that holds it. Do not mix in a second severity scale in the same report.
7. Pilot data is excluded by default. Include it only when the pilot ran the final plan and product version unchanged, and say so in the method note.
8. Keep test data out of production. Use test accounts and seeded data from the plan, and reset them between sessions so each participant starts from the same state.
9. Use the severity tally script below for the report counts. Save it as severity_tally.py, prepare a findings CSV in the format shown there, then run it. Report counts from its output, not from a hand count.
10. No invented claims about tools. Session recording, video calls, and note-taking are described generically in this skill. If a plan names a specific tool, confirm it is available to the team before the first session.

Severity tally (severity_tally.py)

Input format: a CSV file with the header id,task,severity,title and one row per finding. Severity is an integer from 0 to 4 on the scale in Workflow step 8.

Example findings.csv (the fixture used for the expected output below):

csv
id,task,severity,title
F-01,Find a gift under $50,4,Price filter resets after applying it
F-02,Check out with a saved card,4,Pay button gives no response on first tap
F-03,Find a gift under $50,3,Filter panel hides the apply control on small screens
F-04,Change the delivery address,3,Address form rejects a valid apartment number
F-05,Track an existing order,3,Order status page uses an unexplained status term
F-06,Check out with a saved card,2,Saved card list does not show which card is default
F-07,Track an existing order,1,Tracking link is low contrast
F-08,Change the delivery address,0,Participant liked the address autocomplete

Run: python3 severity_tally.py findings.csv

Expected output for that file:

text
total: 8
severity 4: 2
severity 3: 3
severity 2: 1
severity 1: 1
severity 0: 1
findings by severity (highest first):
4 | F-01 | Find a gift under $50 | Price filter resets after applying it
4 | F-02 | Check out with a saved card | Pay button gives no response on first tap
3 | F-03 | Find a gift under $50 | Filter panel hides the apply control on small screens
3 | F-04 | Change the delivery address | Address form rejects a valid apartment number
3 | F-05 | Track an existing order | Order status page uses an unexplained status term
2 | F-06 | Check out with a saved card | Saved card list does not show which card is default
1 | F-07 | Track an existing order | Tracking link is low contrast
0 | F-08 | Change the delivery address | Participant liked the address autocomplete

severity_tally.py (Python 3, standard library only). Save this code as severity_tally.py:

python
#!/usr/bin/env python3
"""Tally usability findings by severity from a CSV. Stdlib only.

Input: CSV file with header id,task,severity,title and one row per finding.
Severity must be an integer from 0 to 4 (this skill's convention).
Usage: python3 severity_tally.py findings.csv
"""
import csv, sys
from collections import Counter

def load(path):
    rows = []
    with open(path, newline="", encoding="utf-8") as f:
        r = csv.DictReader(f)
        need = ["id", "task", "severity", "title"]
        if r.fieldnames is None or any(c not in r.fieldnames for c in need):
            raise ValueError("CSV must have header id,task,severity,title")
        for line_no, row in enumerate(r, start=2):
            raw = (row.get("severity") or "").strip()
            try:
                sev = int(raw)
            except ValueError:
                raise ValueError(f"line {line_no}: severity {raw!r} is not an integer")
            if sev < 0 or sev > 4:
                raise ValueError(f"line {line_no}: severity {sev} is outside 0 to 4")
            rows.append({
                "id": (row.get("id") or "").strip(),
                "task": (row.get("task") or "").strip(),
                "severity": sev,
                "title": (row.get("title") or "").strip(),
            })
    if not rows:
        raise ValueError("no findings")
    return rows

def main():
    if len(sys.argv) != 2:
        print("usage: python3 severity_tally.py findings.csv", file=sys.stderr)
        return 2
    try:
        rows = load(sys.argv[1])
    except (OSError, ValueError) as e:
        print(f"error: {e}", file=sys.stderr)
        return 2
    counts = Counter(r["severity"] for r in rows)
    print(f"total: {len(rows)}")
    for sev in sorted(counts, reverse=True):
        print(f"severity {sev}: {counts[sev]}")
    print("findings by severity (highest first):")
    for r in sorted(rows, key=lambda x: (-x["severity"], x["id"])):
        print(f"{r['severity']} | {r['id']} | {r['task']} | {r['title']}")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())