rubric_design v1.0.0

name: rubric_design description: Design scoring rubrics that different graders apply the same way: choose criteria from the outcomes, write level descriptors that describe the work instead of effort or vague quality, set weights that match importance, and calibrate graders on anchor work before real grading. Use when building or fixing a rubric for essays, projects, presentations, hiring panels, grant or award review, peer assessment, or any task scored by more than one person, and when graders disagree or scores drift.

Purpose

Build a rubric that survives contact with a second grader. A rubric fails in a predictable way: the criteria sound sensible, the level words sound sensible, and two honest graders still land two levels apart on the same work because the cells describe judgments instead of observable features of the work. This skill fixes that at the source, then proves it with a calibration round on real sample work before any live score counts.

Workflow

1. Start from the outcome, not the assignment format

• Write the outcome in one sentence: what should the work show that the person knows or can do.
• List the evidence for that outcome in the work itself. Every criterion in step 2 must trace to a piece of that evidence.
• Separate pass or fail gates from scored criteria. A missing required element, a safety rule, or a deadline policy is a gate or a course rule, not a low score on a quality criterion. Do not let a gate get averaged away, and do not turn compliance checks into fake quality criteria.

1. Choose the rubric type

• Analytic (default): one row per criterion, one score per row. Use it when graders must give diagnostic feedback, when a total can hide a fatal weakness, or when more than one grader scores the same work. It is slower and it is the only type that shows where graders disagree.
• Holistic: one overall score with a paragraph per level. Use it only for fast, low stakes sorting where the whole matters more than the parts and no one needs criterion level feedback.
• Single point: describe only the target level for each criterion, with space on either side for notes on gaps and strengths. Use it when feedback matters more than points and graders are experienced. It does not solve disagreement by itself, so it still needs calibration.

1. Select criteria

• Aim for 3 to 6 criteria, 7 at most. More rows than that and graders stop distinguishing them and learners stop reading them.
• Test each candidate criterion against six checks, drawn from Brookhart: it is appropriate to the outcome, definable in plain words, observable in the work, distinct from the other criteria, part of a set that is complete for the purpose, and describable along a continuum of quality.
• One construct per row. "Evidence and analysis" is two criteria wearing one coat. Split it, or pick the one the outcome actually targets.
• Cut criteria that score effort, time spent, neatness by itself, source counts by themselves, or personality. If a count matters, say what the count buys: not "uses 3 sources" but what those sources do for the claim.
• Run the overlap test: if two rows would almost always get the same score on real work, merge them or rewrite one until they separate.

1. Set the levels

• Use as many levels as graders can tell apart in words, usually 3 to 5, with 4 as the working default. A level you cannot describe as different work is a level that produces random splits.
• Name levels by position, not by praise or blame: 4, 3, 2, 1, or Benchmark, Developing, Proficient, Advanced. Words like Excellent, Good, Fair, and Poor are verdicts, not descriptions, and they belong nowhere in the cells either.
• Decide the zero rule before grading: what gets a 0 (no submission, work that does not attempt the task, work below the lowest described level) and what gets the lowest level instead. Write it on the rubric.
• Write the target level first (the level a competent finished piece should hit), then write the level above it by asking what changes in the work, then back off downward by asking what is missing, weaker, or wrong in the work. Keep the same sentence skeleton across a row so the difference between levels is the work, not the wording.

1. Write descriptors that describe work

• Follow the rules in references/descriptor-writing.md. The short version: describe what is present in the work, in observable terms, with parallel structure across the row, and cover the whole range from the top level to the bottom.
• Ban the empty words from cells: good, strong, weak, thorough, adequate, excellent, poor, and effort words such as tried hard or showed effort. Each one is a place where two graders will fill in different private standards.
• Avoid pure counting as the only difference between levels. "No errors, 1 to 3 errors, 4 to 5 errors" tells a grader nothing about whether the work still reads clearly. Pair any count with the effect on the reader, the result, or the task.
• Check each row with the stranger test: hand one row and one piece of work to someone who did not write the rubric. If they cannot place the work without asking what you meant, the row is not done.

1. Set weights and the scoring rule

• Weight criteria by the importance of the outcome they carry, never by how easy they are to score. Mechanics is easy to count and is still a minor criterion in most writing tasks.
• State weights as percentages that sum to 100, or as points that sum to the stated total. Run bin/rubric_check.py to verify the arithmetic.
• Write the conversion in full: level to points per criterion, weighted total formula, any minimum per criterion (for example, no pass with the core criterion at level 1, whatever the total says), and how a split or missing criterion is handled.
• Worked arithmetic: a criterion weighted 40 percent, scored 3 on a 0 to 4 scale, contributes 40 x 3/4 = 30 points out of 100. Show one worked total on the rubric itself so graders and learners compute it the same way.

1. Calibrate before real grading (non-negotiable)

• Run the full routine in references/calibration-protocol.md. In brief: pick 4 to 6 anchor pieces that span the levels (include one borderline piece on purpose), have every grader score all of them independently and record scores before any discussion, then compare cell by cell.
• Discuss only the cells where graders differ. Each grader points to the words in the work and the words in the descriptor that drove the score. The facilitator records the agreed reading and, where the descriptor caused the split, rewrites that cell on the spot.
• Settle splits with a rule agreed in advance: consensus after discussion for anchors, and for live grading a third grader or the lead adjudicates any piece where two graders differ by more than one level on any criterion or on the total band.
• Measure the round with bin/agreement.py: report exact agreement, agreement within one level, and quadratic weighted kappa per criterion. Report raw agreement next to kappa, because kappa falls when almost all work lands in one level even if graders mostly agree.
• Do not set a universal kappa pass mark. Set the bar from the stakes before the round (higher stakes, tighter bar), record the bar and the result, and if the round misses the bar, fix the descriptors and recalibrate on fresh samples. Rewriting cells and retesting is the normal path, not a failure.
• Recalibrate mid-grading on a large batch: pull a small shared sample, rescore blind, and check for drift and for one grader running systematically high or low.

1. Ship it to learners and graders in usable form

• Give learners the rubric when the task is set, not after grading, and let them score one anchor piece themselves before they start. A rubric first seen on a graded paper only explains a grade.
• Give graders the rubric, the anchors with agreed scores and reasons, the zero rule, the adjudication rule, and one page of scoring notes from calibration. That pack, not the grid alone, is the instrument.

Output Contract

• A rubric in a clean grid or JSON matching assets/rubric-template.json: outcome statement, gates listed separately, 3 to 6 criteria, weights summing to 100, 3 to 5 levels, a descriptor in every cell, a zero rule, and the scoring and conversion rule with one worked total.
• A calibration record: anchor set used, independent scores, exact and adjacent agreement and weighted kappa per criterion from bin/agreement.py, cells rewritten during calibration, and the adjudication rule.
• A grader pack: final rubric, scored anchors with reasons, and calibration notes.
• A learner copy: the same rubric in plain language, shared before the task starts. Do not maintain two different standards, one for learners and one for graders.
• The checker output from bin/rubric_check.py with no errors. Warnings are either fixed or explicitly accepted with a reason in the calibration record.

Operating Rules

• Never ship an uncalibrated rubric for multi-grader use. A grid with no anchor round is a draft, however polished it looks.
• Describe work, not the worker. No cell may score effort, attitude, improvement, or intent, because none of those are visible in the work and every grader invents a different standard for them.
• Fix disagreement in the descriptor first. Grader error is the second hypothesis, checked by whether the same grader also splits on anchors other graders found clear.
• Never change descriptors in the middle of live grading without rescoring the work already graded under the old wording. Note the version on every score sheet.
• Keep facts honest: agreement statistics describe this grader group on this sample. Do not present a calibration result on 5 anchors as proof the rubric will hold on 500 pieces.
• For high stakes decisions (admissions, awards, employment, certification), add blind scoring where practical, double score a stated share of the work, and keep the calibration record with the results. If the stakes are high and no calibration is possible, say so plainly instead of presenting the scores as settled.