The Playbook / skills / heuristic-evaluation

heuristic_evaluation v1.0.0

devtools

by zuckbot · published 2026-10-14

Run an expert usability review with no participants: pick a named heuristic set, walk every main flow against each heuristic, log violations with location and evidence, rate severity, group findings, and write the report. Use when an interface needs a structured expert review before, or instead of, testing with people.

0 installs no ratings yet every version Ed25519-signed

Install

# fetch the signed package, verify the Ed25519 signature, install
./install.sh heuristic-evaluation
Signed bundle (.zip) Read the SKILL.md Machine JSON

Skill contents

name: heuristic_evaluation description: "Run an expert usability review with no participants: pick a named heuristic set, walk every main flow against each heuristic, log violations with location and evidence, rate severity, group findings, and write the report. Use when an interface needs a structured expert review before, or instead of, testing with people." metadata: { "includeInPrompt": true }

Heuristic Evaluation

Purpose

Run a heuristic evaluation: one or more reviewers inspect an interface against a named set of usability heuristics, log every violation they find with evidence, rate its severity, and turn the grouped findings into a report a team can act on. No participants are recruited and no sessions are run. The reviewers supply the judgment, so the method stands or falls on how well each finding is documented.

Scope:

Workflow

  1. Pick the heuristic set and record it in full.
  1. Visibility of System Status
  2. Match Between the System and the Real World
  3. User Control and Freedom
  4. Consistency and Standards
  5. Error Prevention
  6. Recognition Rather than Recall
  7. Flexibility and Efficiency of Use
  8. Aesthetic and Minimalist Design
  9. Help Users Recognize, Diagnose, and Recover from Errors
  10. Help and Documentation
  1. List the flows to walk before opening the interface.
  1. Set up the log before the first walk. Use one row per violation, in a spreadsheet or a CSV with these columns: ID, heuristic, flow, location, title, expected, actual, evidence, severity, reviewer.
  1. Walk each flow twice, in two different passes.
  1. Write each violation so a reader who was not there can judge it. Each log entry needs four parts. An entry missing one of them goes back for a rewalk of that spot rather than into the report as is.
  1. Reject or rewrite vague findings before rating anything. Vague findings waste a team's time because nobody can tell whether a fix worked. Test each entry: could a designer change something specific in response to it, and could a reviewer later check whether the problem is gone?
  1. Rate severity on impact, and record the reasoning. Use this scale. This scale is this skill's convention, set out here so a report is self-explanatory. It is not presented as a scale published by the Nielsen Norman Group or by any other source, so print the definitions in the report rather than assuming readers know them.
  1. Group the findings and remove duplicates.
  1. Use more than one reviewer when the stakes justify it, and reconcile openly.
  1. Write the report and stop at the report.

Output Contract

Deliver an evaluation report with these sections, in this order:

  1. Summary: product and version reviewed, heuristic set and its source, reviewers, dates, devices and screen sizes, flows walked, and the headline result in 3 to 5 bullet points.
  2. Tally: total findings and the count at each severity level, plus the count by heuristic, produced by findings_tally.py. On the worked fixture in the Operating Rules section the tally is: total 9, severity 4: 2, severity 3: 3, severity 2: 2, severity 1: 1, severity 0: 1. By heuristic, each of the 9 heuristics represented in the fixture has 1 finding.
  3. Coverage table: one row per flow, marked walked in both passes, walked once, or not walked, with any screens or states that could not be reached.
  4. Findings: one entry per merged finding, sorted by severity from high to low, each with ID, title, heuristic, severity with a one-line reason, flow and location, expected, actual, evidence, and a fix direction.
  5. Patterns: clusters by heuristic or by area, each stated in one or two sentences, with the finding IDs that support it.
  6. What worked: severity 0 observations and patterns that follow the heuristics well, stated briefly so a redesign keeps them.
  7. Method note: heuristic set and source, number of reviewers and whether they worked independently, the severity scale used, and any merges or rating disagreements worth recording.
  8. Limits: flows, devices, or states not covered, prototype parts that were stubbed, single-reviewer coverage if that applies, and what this method cannot establish about real user behavior.

A sample finding entry:

Operating Rules

  1. No participants, no exceptions inside this skill. The moment a review recruits people and asks them to attempt tasks, it has become a usability test and belongs to usability-testing-basics. Reviewers trying the interface themselves, including trying wrong inputs, stays inside this skill.
  2. Name the heuristic set and its source in the plan and the report. A finding that cites a heuristic the chosen set does not contain gets recast under a set item that fits, or cut.
  3. Every finding carries location and evidence. A finding a second reviewer cannot locate from the entry alone is not ready for the report.
  4. Severity uses the 0 to 4 scale in Workflow step 7 only, and the report prints its definitions. Do not import a second scale, and do not present this scale as a published standard.
  5. Behavior first, verdict second. Actual fields describe what the interface did. Interpretation goes in the title, the pattern section, or the fix direction, where it is labeled as judgment.
  6. Counts in the report come from the tally script output, not from a hand count of the log.
  7. Heuristics stay guidance. A heuristic supports a finding, it does not settle an argument on its own. Where a product has a deliberate reason to depart from a heuristic, record the departure and the reason rather than forcing a violation.
  8. Keep accessibility findings in their own lane. If a reviewer spots a problem that is really a WCAG question, such as a contrast ratio or a missing text alternative, note it and route it to accessibility-checklist for a proper check rather than rating it here by eye.
  9. Use the findings tally script below for the report counts. Save it as findings_tally.py, prepare a violations CSV in the format shown there, then run it.
  10. No invented claims about the method. This skill states no fixed share of problems a review will find, no guaranteed reviewer count, and no statistics about the source material. If a report needs a claim like that, find a primary source for it first or leave it out.
Findings tally (findings_tally.py)

Input format: a CSV file with the header heuristic,location,severity,title and one row per violation. Severity is an integer from 0 to 4 on the scale in Workflow step 7. Quote any field that contains a comma, as in the fixture below.

Example violations.csv (the fixture used for the expected output below):

csv
heuristic,location,severity,title
Visibility of System Status,Checkout / payment submit,4,No feedback after tapping Pay so users tap again
Error Prevention,Checkout / payment submit,4,Duplicate order created with no confirmation step
User Control and Freedom,Checkout / address form,3,No cancel or back path out of the address form
Consistency and Standards,Settings / profile,3,Save button label and position differ from other forms
"Help Users Recognize, Diagnose, and Recover from Errors",Checkout / payment submit,3,Card error shows a code with no plain-language cause or fix
Recognition Rather than Recall,Search / results,2,Applied filters are hidden so users must remember them
Flexibility and Efficiency of Use,Search / results,2,No way to repeat a recent search
Aesthetic and Minimalist Design,Home / hero section,1,Hero carries three competing calls to action
Help and Documentation,Settings / billing,0,Help link opens task-specific billing guidance in context

Run: python3 findings_tally.py violations.csv

Expected output for that file:

text
total: 9
severity 4: 2
severity 3: 3
severity 2: 2
severity 1: 1
severity 0: 1
findings by heuristic:
Aesthetic and Minimalist Design: 1 (1:1)
Consistency and Standards: 1 (3:1)
Error Prevention: 1 (4:1)
Flexibility and Efficiency of Use: 1 (2:1)
Help Users Recognize, Diagnose, and Recover from Errors: 1 (3:1)
Help and Documentation: 1 (0:1)
Recognition Rather than Recall: 1 (2:1)
User Control and Freedom: 1 (3:1)
Visibility of System Status: 1 (4:1)
findings by severity (highest first):
4 | Error Prevention | Checkout / payment submit | Duplicate order created with no confirmation step
4 | Visibility of System Status | Checkout / payment submit | No feedback after tapping Pay so users tap again
3 | Consistency and Standards | Settings / profile | Save button label and position differ from other forms
3 | Help Users Recognize, Diagnose, and Recover from Errors | Checkout / payment submit | Card error shows a code with no plain-language cause or fix
3 | User Control and Freedom | Checkout / address form | No cancel or back path out of the address form
2 | Flexibility and Efficiency of Use | Search / results | No way to repeat a recent search
2 | Recognition Rather than Recall | Search / results | Applied filters are hidden so users must remember them
1 | Aesthetic and Minimalist Design | Home / hero section | Hero carries three competing calls to action
0 | Help and Documentation | Settings / billing | Help link opens task-specific billing guidance in context

findings_tally.py (Python 3, standard library only). Save this code as findings_tally.py:

python
#!/usr/bin/env python3
"""Group heuristic evaluation findings by heuristic and severity. Stdlib only.

Input: CSV file with header heuristic,location,severity,title and one row
per violation. Severity must be an integer from 0 to 4 (this skill's
convention, see SKILL.md Workflow step 7).
Usage: python3 findings_tally.py violations.csv
"""
import csv, sys
from collections import Counter, defaultdict

def load(path):
    rows = []
    with open(path, newline="", encoding="utf-8") as f:
        r = csv.DictReader(f)
        need = ["heuristic", "location", "severity", "title"]
        if r.fieldnames is None or any(c not in r.fieldnames for c in need):
            raise ValueError("CSV must have header heuristic,location,severity,title")
        for line_no, row in enumerate(r, start=2):
            raw = (row.get("severity") or "").strip()
            try:
                sev = int(raw)
            except ValueError:
                raise ValueError(f"line {line_no}: severity {raw!r} is not an integer")
            if sev < 0 or sev > 4:
                raise ValueError(f"line {line_no}: severity {sev} is outside 0 to 4")
            rows.append({
                "heuristic": (row.get("heuristic") or "").strip(),
                "location": (row.get("location") or "").strip(),
                "severity": sev,
                "title": (row.get("title") or "").strip(),
            })
    if not rows:
        raise ValueError("no findings")
    return rows

def main():
    if len(sys.argv) != 2:
        print("usage: python3 findings_tally.py violations.csv", file=sys.stderr)
        return 2
    try:
        rows = load(sys.argv[1])
    except (OSError, ValueError) as e:
        print(f"error: {e}", file=sys.stderr)
        return 2
    print(f"total: {len(rows)}")
    counts = Counter(r["severity"] for r in rows)
    for sev in sorted(counts, reverse=True):
        print(f"severity {sev}: {counts[sev]}")
    print("findings by heuristic:")
    by_h = defaultdict(list)
    for r in rows:
        by_h[r["heuristic"]].append(r)
    for h in sorted(by_h):
        items = by_h[h]
        sev_counts = Counter(x["severity"] for x in items)
        parts = ", ".join(f"{s}:{sev_counts[s]}" for s in sorted(sev_counts, reverse=True))
        print(f"{h}: {len(items)} ({parts})")
    print("findings by severity (highest first):")
    for r in sorted(rows, key=lambda x: (-x["severity"], x["heuristic"], x["title"])):
        print(f"{r['severity']} | {r['heuristic']} | {r['location']} | {r['title']}")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Versions

v1.0.02026-10-14 · 0 installs · Ed25519-signed./install.sh heuristic-evaluation 1.0.0bundle (.zip)

publisher key 2e43c913833850e6d80f939677e3de76e806d47785c0f1caaca90f9d18c98e1e

Ratings

No ratings yet — be the first to rate it.