reader_panel_testing v1.0.0

name: reader_panel_testing description: Test a draft with a small panel of independent readers who receive it in chunks, mark the exact point where clarity or attention drops, and score each chunk, then turn the marks into a ranked revision list with a protect list and a retest plan. Use when someone asks to test a draft on real readers, run a beta read, find where readers get confused, bored, or stop, check whether a chapter, guide, essay, or story lands with its target audience, or turn scattered reader feedback into a revision order. For a single expert diagnosis of a story use a story diagnosis skill; for product usability tasks use a usability testing skill.

Reader Panel Testing

Purpose

Find the exact places a draft loses its readers, while the readers are still in the act of reading. End of draft feedback fails in predictable ways. Readers forget where they stumbled, later chapters quietly repair earlier confusion in their memory, and polite general praise replaces the one useful fact, which is where the text stopped working. This skill fixes the timing. Readers get the draft in chunks, in order, with no way to read ahead. They mark the spot the moment clarity or attention drops. They say back what each chunk said before they see the next one. The author gets locations, counts, and scores instead of impressions, and revises in the order the evidence demands.

The panel is run by one person with ordinary tools: email or messages, a shared folder, and the files in this skill. No lab, no facilitator team, no group session. It works for fiction chapters, guides, essays, reports, course material, and any draft whose job is to be understood and finished by a target audience.

This is reader response, not editing and not expert review. Readers report what happened in their heads. They never fix the text, grade it, or argue about what it should have been. Peter Elbow's reader based feedback tradition calls this giving the writer the movies of the reader's mind. The Federal Plain Language Guidelines build document testing on the same rule: the tester records what the reader took the text to mean and does not correct them, and wherever readers misunderstood, the document has the problem.

Workflow

1. Write the intended point of each chunk before anyone reads. One sentence per chunk: what a reader should carry away from it. For fiction, write what should have changed, landed, or become clear. Seal these sentences until scoring time. Without them there is no honest way to grade the say backs, and authors rewrite their intentions to match whatever readers happened to take away.

1. Recruit 3 to 5 readers from the real target audience. Follow the recruiting rules in references/panel-protocol.md. The short version: target readers, not fellow writers and not subject experts; assign IDs R1 to R5; agree the deadline and any payment or payback before chunk 1 goes out. Two readers is a minimum viable run and gets labelled as such in the report. One reader is a single read, not a panel. For a high stakes public or legal document, skip this lighter method and run formal paraphrase testing at the scale the Federal Plain Language Guidelines describe, 6 to 9 one on one interviews per document.

1. Cut the draft into numbered chunks. Cut at natural breaks, number every paragraph in the copies readers receive, and freeze the draft for the whole run. Starting sizes and the short piece cue method are in references/panel-protocol.md. A chunk a reader cannot finish in one sitting is too long for this test, because a natural overnight stop then looks exactly like a drop point.

1. Send the reader brief and chunk 1, and nothing else. Use assets/reader-brief.md unchanged for every reader. Send no outline, no summary, no explanation of what the draft is trying to do, and no earlier feedback from anyone. What the author explains in advance, the panel can no longer test.

1. Collect marks in the moment, with the six codes. Readers mark as they read: X for stopped or would stop, ? for confused or lost, R for had to reread, S for skimmed or attention dropped, Q for an open question they are carrying, and + for a passage that hooked them. Each mark gets a location and one short first person note. Readers do not explain causes and do not propose fixes. Career Authors' beta reader guidance gives the same instruction: flag where you skimmed, where you were confused, and where you had to back up, without carrying the burden of saying why or how to fix it. This adapts the think aloud method from usability and reading research into writing, because a panel runs async and the mark sheet is where the thinking gets recorded.

1. Take the say back and the two scores at the end of every chunk. Before a reader sees the next chunk, they fill in the say back on assets/mark-sheet.md: what the chunk said in their own words, what they expect next, and the one point they would repeat to a friend. Then clarity 1 to 5 and pull 1 to 5. This is the cue method from federal paraphrase testing, where the reader stops at a set point and says what the section means in their own words. A reader who cannot say back what a chunk said has told you the chunk failed, whatever their marks say.

1. Release chunks one at a time, per reader, only against returned marks. A reader gets chunk N when their marks and say back for chunk N minus 1 are in hand. Never let a reader read ahead, never discuss the draft with readers mid run, and never let readers see each other's marks before all marks are in. Reading is a solo act, and group settings corrupt the data: the Government of Canada's document usability guidance warns that readers in a group may not admit they are lost and will work together to figure the text out, which no lone reader at home can do. A reader who wants to quit, quits. Their last mark is X at the place they stopped, and that is first class data, not a failed recruit.

1. Transcribe the marks and score the run. Put every mark into a CSV with columns reader, chunk, location, code, note, and every chunk score into a second CSV with reader, chunk, clarity, pull, paraphrase. Grade each say back ok, partial, or off against the sealed intended point from step 1, using the grading rules in references/panel-protocol.md. Then run python3 ~/workspace/skills/reader-panel-testing/bin/panel_score.py marks.csv --scores scores.csv --out panel-report.md. The scorer ranks chunks by risk, lists hot spots where at least half the readers left a negative mark at the same location, and lists chunks to protect. Risk per chunk is weighted marks per reader (X counts 3, ? and R count 2, S and Q count 1, + counts 0) plus the clarity gap (5 minus mean clarity) plus the pull gap (5 minus mean pull) plus the paraphrase penalty (off 2, partial 1, ok 0, averaged). Bands: P1 at 5.0 or more, P2 from 2.5 to 4.99, P3 below 2.5.

1. Turn the report into a ranked revision list. Use assets/revision-list-template.md. Order of work: hot spots first, in risk order; then chunks with off say backs, because a reader taking away the wrong point is a structure failure, not a style failure; then P2 chunks; then cheap P3 fixes. Every item names the evidence, a one sentence diagnosis, the fix the author will make, and the retest plan. Single reader marks go on the watch list, not the revision list, unless the mark reports a factual error, a safety or legal problem, or the reader is the one panel member squarely inside the target audience. Write down any panel finding the author chooses not to act on, with the reason. A worked example with real scorer output and a before and after rewrite of the worst paragraph is in references/worked-example.md.

1. Protect what worked while you revise what did not. Any passage with + marks from two or more readers goes on the protect list. It does not get cut, shortened, or smoothed down as a side effect of fixing its neighbours. Revision that fixes the weak chunks by flattening the strong ones trades one problem for another, and the panel cannot warn you, because the panel tested the old draft.

1. Retest the changed chunks. Testing once is a snapshot; the federal guidelines treat test, revise, and retest as one cycle and say to plan on testing at least twice. Send each heavily revised chunk to at least two readers, with at least one reader who has never seen any version of the draft. A fresh reader is the only clean check, because a returning reader now reads the fix with the old version and the author's intentions in their head. Score the retest the same way, compare risk against the first run, and close a revision item only when its hot spot clears and its say backs come back ok.

1. Close the run in writing. File the raw mark sheets, both CSVs, the scorer report, and the revision list together, with the draft version named on each. Thank the readers and tell them, in two or three lines, what their marks changed. Readers who hear what their work did give honest marks again. Readers who hear nothing give polite marks next time, and a polite panel is a wasted panel.

Output Contract

Deliver, in this order:

1. The panel report from bin/panel_score.py: readers and count, ranked chunk table with bands and risk scores, hot spot list, and protect list.
2. The ranked revision list on the template: every item with its evidence, diagnosis, planned fix, and retest plan, plus the watch list and any decisions made against the panel, with reasons.
3. The retest result for every P1 item: new risk score beside the old one, the fresh reader's say back grade, and whether the item is closed or goes back for another pass.
4. The raw inputs, kept with the report: the mark CSV, the scores CSV, the sealed intended points, and the reader brief as sent.

Operating Rules

• Never explain the draft to a reader before or during the run. No intent, no backstory, no "what this part is really about." Every explanation destroys the exact measurement the panel exists to make.
• Never argue with a mark. A reader cannot be wrong about where they got lost, skimmed, or stopped. They can only be one reader, which the counts and the watch list already handle.
• Never show readers each other's marks, scores, or comments before every mark for every chunk is in. Independence is the whole value of a panel over a conversation.
• Keep readers reporting and the author deciding. Readers do not rewrite sentences, vote on endings, or settle disagreements between themselves. The author owns every fix and records every override.
• Do not use the panel as a copyedit, a fact check, or a sensitivity review. Those are different jobs with different readers. A panel can surface a factual error a reader trips on, and that error gets fixed, but a clean panel run never certifies accuracy.
• Report the denominators every time. A hot spot is 3 of 4 readers, a risk score sits beside the reader count, and a two reader run is labelled a two reader run in every document it produces.
• Do not invent marks, scores, or reader comments to fill a thin run. If only two readers finished, the report says two readers finished. If a chunk came back with no marks at all, report that as a finding to interpret with care, not as proof the chunk is perfect.
• Keep reader identities out of the report. IDs only. Store names and contact details in the recruiting note, apart from the files the author will argue with.