interview_transcript_analysis v1.0.0

name: interview_transcript_analysis description: "Code qualitative interview transcripts: build a codebook, extract themes, select quotes with context, check saturation, and measure coder agreement. Use when analyzing interview or focus group transcripts after the interviews are done." metadata: { "includeInPrompt": true }

Interview Transcript Analysis

Purpose

Turn a set of qualitative interview transcripts into a defensible analysis: a clear codebook, coded segments, themes supported by quotes in context, a saturation check, and a record of how coders resolved disagreements.

This skill is for analysis after data collection. It does not cover how to recruit participants, write an interview guide, or conduct the interview itself.

Not this skill:

• customer-interviews: conducting product discovery interviews and live note-taking, not coding transcripts after the fact.
• survey-data-analysis: quantitative analysis of closed survey responses, not qualitative text coding.
• meeting-transcript-coaching: coaching a speaker's delivery from meeting transcripts, a different goal and setting.
• lit-review-synthesis: synthesizing published literature, not primary interview transcripts.

Workflow

1. Prepare the transcripts.

• Confirm consent and anonymization status before coding. Replace names, organizations, and identifying details with stable pseudonyms and keep the mapping in a separate, restricted file.
• Keep one transcript per participant, with a participant ID, date, and interview length. Preserve speaker turns and timestamps where they exist.
• Define the coding unit before coding starts. Common choices are a meaning unit (a passage expressing one idea, which can span a few sentences) or a speaker turn. Pick one, write it down, and use it consistently. Exact unit boundaries affect agreement scores, so do not change the unit mid-project without recoding.

1. Build the codebook.

• Start with deductive codes where they are justified: codes derived in advance from the research questions, prior theory, or an existing framework the project has committed to. List only codes the project can point to a source for.
• Add inductive codes from the data: read a small first set of transcripts (often 2 to 3, or about 10 to 20 percent of the set) without forcing them into the deductive list, and note concepts participants raise in their own terms.
• Write each code as one entry with: code name, a one or two sentence definition, when to use it, when not to use it, and one short example quote from the data. The "when not to use it" line is what keeps similar codes separate.
• Pilot the draft codebook on one or two transcripts, revise definitions where coders hesitate or overlap, and freeze a version number before full coding. Record later changes in a change log with the date and the reason, and recode earlier material if a definition changed in substance.

1. Code the full set.

• Code every transcript against the frozen codebook. A segment can carry more than one code if it genuinely covers two ideas; note that choice in the codebook so all coders do it the same way.
• Write a short memo per transcript: main points, surprises, anything that contradicts earlier interviews, and candidate new codes. Memos are where inductive codes and later themes start.
• If a passage fits no code, mark it as uncoded and review the pile regularly. A growing uncoded pile usually means a missing code. A code applied to almost everything usually means its definition is too broad.

1. Check coder agreement and handle disagreement.

• Where two or more coders are involved, have them code the same subset independently before discussing it. A subset of about 10 to 25 percent of units is a common working choice, but the right size depends on the study; state the choice and the reason in the report instead of treating any percentage as a rule.
• Report two numbers: percent agreement (units agreed on divided by total units, times 100) and Cohen's kappa, which adjusts observed agreement for the agreement expected by chance given each coder's category distribution.
• Formula for two coders: kappa = (Po - Pe) / (1 - Pe), where Po is the observed proportion of agreement and Pe is the chance-expected agreement, computed as the sum over categories of (coder 1's share in that category times coder 2's share in that category).
• Worked example, verified by hand and with the script below: 20 units, two codes A and B. Both coders chose A on 8 units, both chose B on 7 units, coder 1 chose A while coder 2 chose B on 3 units, and the reverse on 2 units. Po = 15 / 20 = 0.75. Coder 1 used A 11 times and B 9 times; coder 2 used A 10 times and B 10 times. Pe = (11/20 x 10/20) + (9/20 x 10/20) = 0.275 + 0.225 = 0.50. Kappa = (0.75 - 0.50) / (1 - 0.50) = 0.50. Percent agreement = 75.0 percent.
• Use the script in the Operating Rules section to compute both numbers from a CSV of paired codes.
• Treat disagreement as information, not as an error to hide. For each disagreement, classify the cause: vague code definition, a genuinely ambiguous passage, different unit boundaries, or a simple slip. Fix definitions for the first cause, record a decision rule for the second, realign boundaries for the third, and correct slips without changing the codebook.
• After discussion, record the consensus code for reporting, but keep the original independent codes and the pre-discussion agreement scores in the project record. Do not report only post-discussion agreement; that overstates reliability.
• Kappa has known limits: it falls when one code dominates the data even if raw agreement is high, and it is designed for two coders assigning one category per unit in this simple form. If the coding scheme allows multiple codes per unit or more than two coders, say so and report agreement per code (percent agreement per code is a plain, honest option) instead of forcing this formula onto data it does not fit. Published label bands for kappa ranges exist, but they are conventions that vary by field, so report the number, the category distribution, and the sample size rather than resting a conclusion on a single label.

1. Extract themes.

• Group codes into candidate themes by looking for a shared central idea, not just a shared topic. A topic summary ("what people said about onboarding") is weaker than a theme that makes a claim about the data ("new users trusted the tool only after a coworker vouched for it").
• Test each candidate theme against the coded data: does it hold across participants, does it have enough supporting segments to be more than one person's view, and are there cases that contradict it? Keep a note on disconfirming cases and address them in the write-up instead of dropping them.
• Merge themes that make the same point and split themes that bundle two different claims. Name themes in short, plain phrases that state the point.

1. Select quotes with context.

• For each theme, pick quotes that show the point directly, drawn from more than one participant where the theme claims breadth.
• Include enough surrounding text for the quote to be understood on its own: who is speaking in general terms (participant ID and a relevant, non-identifying descriptor such as role or experience level, where consent allows it), what question or moment prompted it, and the sentences before or after that qualify it.
• Do not trim a quote in a way that changes its meaning. Mark omissions inside a quote with an ellipsis, keep edits to grammar minimal, and note any cleaning in the methods section.
• Check each selected quote against the transcript. A quote that cannot be traced to a transcript location (participant, timestamp or line reference) does not go in the report.

1. Check saturation.

• Track, in coding order, when each new code and each new theme-level insight first appears. A simple table of interview number against new codes found is enough.
• A common working definition: saturation is approached when successive interviews add no new codes and no material change to existing themes. State the project's own stopping criterion before or early in analysis, then report whether it was met and at which interview the count of new codes levelled off.
• Report saturation honestly. If interviews were fixed in advance by budget or a course deadline, say the set size was set in advance and report what the new-code curve showed, rather than claiming saturation was the reason data collection stopped. Saturation is a judgment about this dataset and these research questions, not a general guarantee.

1. Write up the analysis.

• Follow the Output Contract below. Lead with the themes and the evidence, and keep the methods detail (codebook version, unit, agreement scores, saturation table) in the report so a reader can judge how the claims were produced.

Output Contract

Deliver, in this order:

1. Summary: the research questions, the number of transcripts and participants in general terms, and the headline themes in 3 to 6 bullet points.
2. Codebook: a table with code name, definition, when to use, when not to use, and one example quote per code, plus the codebook version and change log.
3. Theme report: one section per theme with the theme claim, a short explanation, the supporting codes, how widely it appeared (for example, in how many of the transcripts), disconfirming cases, and 1 to 3 traced quotes with context.
4. Agreement report (if more than one coder): number of double-coded units, percent agreement, Po, Pe, kappa with the formula used, the main causes of disagreement, and what was changed in the codebook or decision rules as a result.
5. Saturation note: the new-codes-per-interview table or summary, the stopping criterion the project set, and a plain statement of whether it was met.
6. Limitations: small or narrow sample, single coder if applicable, codes that stayed rare or contested, and anything the data cannot support.

Counts in the theme report describe this dataset only. Phrasing such as "7 of 12 participants" is fine; converting those counts into population percentages or general claims about all users is not supported by this kind of data.

Operating Rules

1. Keep the audit trail. Codebook versions, coding files, memos, agreement calculations, and the quote-to-transcript references stay together in the project folder. A claim that cannot be traced back through that trail gets removed or softened.
2. Anonymize before sharing. Quotes and codebook examples carry the same anonymization as the transcripts. Check quotes again at write-up time, since a harmless detail in a transcript can identify someone when quoted alone.
3. One coder is a limitation, not a disqualification. If only one person coded, say so plainly, use the pilot-and-revise step and regular re-reads of early transcripts as the consistency check, and skip kappa rather than inventing a second set of codes after discussion.
4. Do not code from summaries alone when transcripts exist. Summaries and AI-generated digests can help navigation, but coding decisions and quotes come from the transcript text itself.
5. Keep deductive and inductive codes labelled by origin in the codebook. Mixing them without a label hides which claims were tested against the data and which came from it.
6. Report agreement before consensus discussion, and keep the independent codes. Recomputed agreement after coders have talked is not a reliability measure.
7. Use the agreement calculator in the subsection below for two coders and one code per unit. Save it as kappa.py, prepare a ratings CSV in the format shown there, then run it.
8. No invented saturation or reliability claims. If a check was not run, the report says it was not run. Plain statements about what was actually done are the standard for this skill.

Agreement calculator (kappa.py)

Input format: a CSV file with the header coder1,coder2 and one row per coded unit, holding the code each coder assigned.

Example ratings.csv for the worked example in step 4 of the Workflow:

csv
coder1,coder2
A,A
A,A
A,A
A,A
A,A
A,A
A,A
A,A
B,B
B,B
B,B
B,B
B,B
B,B
B,B
A,B
A,B
A,B
B,A
B,A

Run: python3 kappa.py ratings.csv

Expected output for that file:

text
n: 20
agreements: 15
percent_agreement: 75.0%
po: 0.7500
pe: 0.5000
kappa: 0.5000

kappa.py (Python 3, standard library only). Save this code as kappa.py:

python
#!/usr/bin/env python3
"""Percent agreement and Cohen's kappa for two coders. Stdlib only.

Input: CSV file with header coder1,coder2 and one row per coded unit.
Usage: python3 kappa.py ratings.csv
"""
import csv, sys
from collections import Counter

def compute(pairs):
    if not pairs:
        raise ValueError("no pairs")
    n = len(pairs)
    agree = sum(1 for a, b in pairs if a == b)
    po = agree / n
    c1 = Counter(a for a, _ in pairs)
    c2 = Counter(b for _, b in pairs)
    cats = set(c1) | set(c2)
    pe = sum((c1.get(c, 0) / n) * (c2.get(c, 0) / n) for c in cats)
    if pe == 1.0:
        kappa = 1.0 if po == 1.0 else 0.0
    else:
        kappa = (po - pe) / (1.0 - pe)
    return {"n": n, "agree": agree, "percent_agreement": po * 100.0, "po": po, "pe": pe, "kappa": kappa}

def main():
    if len(sys.argv) != 2:
        print("usage: python3 kappa.py ratings.csv", file=sys.stderr)
        return 2
    pairs = []
    with open(sys.argv[1], newline="", encoding="utf-8") as f:
        r = csv.DictReader(f)
        if r.fieldnames is None or "coder1" not in r.fieldnames or "coder2" not in r.fieldnames:
            print("error: CSV must have header coder1,coder2", file=sys.stderr)
            return 2
        for row in r:
            a = (row.get("coder1") or "").strip()
            b = (row.get("coder2") or "").strip()
            if a or b:
                pairs.append((a, b))
    try:
        out = compute(pairs)
    except ValueError as e:
        print(f"error: {e}", file=sys.stderr)
        return 2
    print(f"n: {out['n']}")
    print(f"agreements: {out['agree']}")
    print(f"percent_agreement: {out['percent_agreement']:.1f}%")
    print(f"po: {out['po']:.4f}")
    print(f"pe: {out['pe']:.4f}")
    print(f"kappa: {out['kappa']:.4f}")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())