--- name: oral_examination description: Designing and running oral examinations that measure understanding instead of recitation: question ladders that climb from recall to transfer, prepared follow-up probes that separate memorised lines from real grasp, bias controls for a live face to face format, and rubric scoring with independent second examiner checks and calibration. Use when building or fixing a viva, oral exam, thesis defence, language or clinical oral, or any spoken assessment scored live by examiners, and when oral scores disagree, drift, or look open to bias or appeal. --- # Purpose Make an oral exam test what the candidate understands, not how well they can recite or perform. The unstructured oral has known faults: examiners lean on recall, time and case difficulty vary from candidate to candidate, and first impressions move scores. A 2023 systematic review and meta-analysis in BMC Medical Education (Abuzied and Nabag, 24 studies) found structured viva reliability at Cronbach alpha 0.80 and 0.75 in two settings against 0.50 for a traditional viva in the comparison reported, with high heterogeneity between studies, so treat that as support for structure, not as a promise. This skill gives the structure: a ladder of questions per topic, probes written in advance, scoring against descriptors, calibration on anchors before the sitting, and a second examiner check that happens on independent sheets, not in a chat after the fact. # Workflow 1. Decide what the oral is for, and what it is not for - Write the outcome in one sentence per topic: what a passing candidate can do with the material, not only say about it. If the honest outcome is recall only, use a written recall test instead. An oral earns its cost when it can follow an answer and find out what sits behind it. - Fix the format before any questions: length, number of topics, who leads, what materials the candidate may use, how the decision is made, and what happens if the exam has to stop. Hold the format identical for every candidate in the sitting. - Tell candidates the format in writing in advance, and offer a mock oral where the stakes allow it. An unfamiliar format taxes nerves, not knowledge, and the first candidate of the day should not be the one who pays for that. 2. Build one question ladder per topic - Follow references/question-ladders.md and start from assets/ladder-template.json. Each ladder climbs the revised Bloom levels on one case: Remember, Understand, Apply, Analyze, Evaluate, then Transfer at the top of a short ladder. - Write a credit line for every rung before the exam: what an answer must contain to earn that rung. Write the floor question, the prepared probes, and the stop rule on the same sheet. - Write a parallel ladder at matched difficulty for the next candidate, room, or sitting, and have a colleague who did not write them check the pair for a rung that is plainly harder in one form. - Keep recall a small share of the total score. A ladder that stops at the second rung for most candidates is telling you the teaching or the ladder is off. Record which it is, do not just lower the cut in the room. 3. Attach probes to the rungs - Use the five probes in references/probe-bank.md: why, example, source, change, and boundary. Write the probe next to the rung it tests, word for word. - Probe a polished answer once before scoring it high, and probe a rough answer once before scoring it low. Polish and understanding are different things, and so are hesitation and ignorance. - Cap it at one probe per answer by default, two at most, and keep the wording neutral. Write the probe used in the notes beside the answer it produced. 4. Write the scoring rubric for live use - Use 3 to 5 criteria tied to the outcomes, for example understanding, reasoning, application and transfer, and evaluation. Weights sum to 100. Use the sheet format in assets/score-sheet.json. - Write descriptors that describe what the candidate said at each level, 0 to 4, in observable terms. Ban verdict words from cells: good, strong, weak, impressive. Each one hands the score back to private taste. - Score communication only where it is part of the stated outcome, and then define it as whether the reasoning could be followed. Accent, confidence, eye contact, and polish are not criteria. See references/bias-and-fairness.md. - Each examiner also gives a global rating, clear fail, borderline, clear pass, or a finer ordered scale, formed independently of the criterion total. That independence is what later lets the rating set or check the cut score. 5. Set the pass standard before anyone is examined - Pick one method and record it: a fixed criterion cut tied to the pass level descriptors, the borderline group method, or borderline regression. The last two are set out in references/calibration-protocol.md, with their limits. - There is no universal pass percentage and no universal agreement bar in this skill, on purpose. Both belong to the institution and the stakes. Write the chosen cut, or the method that will produce it, on the exam record before the sitting starts. - Define the borderline band around the cut in advance. Candidates inside the band get targeted double scoring and first call on adjudication time, because that is where a wrong pass or fail is most likely. 6. Calibrate the examiners on anchors - Run the anchor round in references/calibration-protocol.md before the sitting: three to five recorded, transcribed, or scripted performances spanning the levels, one of them borderline on purpose, scored independently by every examiner. - Measure the round with bin/oral_score.py in calibration mode: exact agreement, agreement within one level, and quadratic weighted kappa, per criterion and overall, reported together. - Discuss only splits over one level, rewrite descriptors that caused splits, and recalibrate on fresh anchors if the round misses the bar the board set in advance. No examiner scores live candidates on a rubric they have not calibrated on. 7. Run the sitting - One examiner leads a ladder while the other observes and writes notes, then swap ladders. Notes quote or closely summarise what the candidate said, rung by rung, with probes used marked. - Work the stop rule: two rungs in a row with no credit, even after one neutral prompt each, and you drop a rung, confirm the floor, and close the ladder. Do not spend the remaining time pressing a stuck candidate. - Close every oral the same way: one final transfer question if the ladders left the top untested, then ask the candidate whether there is a point they expected to cover that the exam missed, and let them add it. Record anything that changes a score. - Handle illness, distress, or a broken adjustment by pausing or stopping and recording the time and point reached. Never score distress as weak knowledge. 8. Run the second examiner check - Both examiners submit independent sheets first. Then run bin/oral_score.py on the pair. It validates weights and levels, computes both weighted totals, and flags criterion splits over one level, a total split over 10 points, differing global ratings, a pass and fail split at the cut, and totals inside the borderline band. - AGREE: record the result under the aggregation rule written in advance, and keep both sheets. - DISCUSS: discuss only the flagged criteria, each examiner quoting candidate words and descriptor words, rescore those criteria independently, then agree. Record what changed and why. - ADJUDICATE: never average a pass and a fail into a pass. A third examiner scores the recording or full notes blind, or the named lead decides under the prewritten rule. Record the adjudicator, the evidence, and the reason. If no recording exists and the notes cannot settle it, say so and follow the institution route for a fresh oral. - On a long day, run the midpoint drift check from references/calibration-protocol.md, and review the whole sitting spread per examiner and per ladder before results go out. 9. Give feedback that matches the evidence - Feedback names the rung the candidate reached per topic, one thing the probing showed they understood well, and the first rung where answers thinned out, with an example of what a stronger answer would have added. - Never feed back delivery comments dressed as academic judgement. If communication was not a scored criterion, it does not appear in the feedback either. # Output Contract - An exam plan: outcomes per topic, format, timing pattern held constant across candidates, candidate facing format note, adjustments agreed, and the pass standard method with the borderline band, all written before the sitting. - A ladder pack: one ladder per topic plus a parallel form, each with stem, rungs with credit lines, floor question, prepared probes, and stop rule, matching assets/ladder-template.json. - A scoring pack: rubric with 3 to 5 weighted criteria and level descriptors, score sheets matching assets/score-sheet.json, and the global rating scale. - A calibration record: anchors used, independent scores, agreement and kappa output from bin/oral_score.py, descriptors rewritten, the bar set and whether the round met it. - A sitting record per candidate: notes by rung with probes marked, both independent sheets, the checker output and verdict, any discussion or adjudication with reasons, and any incident. - A worked example for training new examiners: references/worked-examination.md shows one full ladder, two candidates graded against it, and why the reciter scores lower than the untidy thinker. # Operating Rules - Same ladders, probes, time, and sheet for every candidate in a sitting. Variation the examiner introduces is where most unfairness enters an oral. - Score the answer the candidate gave, at the rung it actually reached. A perfect recitation on an Analyze question earns the lower rung credit, and a rough answer that survives probing earns the higher one. - Independent first, discussion second, always. A second examiner who has heard the first examiner view before scoring is not a check, they are an echo. - Never average away a pass and fail split, a missing criterion, or a global rating disagreement. Each has a named route in step 8. Use it and record it. - Keep factual claims honest. Structure is supported by the evidence as improving reliability and reducing bias risk. It does not remove examiner variability, and this skill states no universal cut score, timing, or agreement threshold. - The notes are the exam. An oral leaves no script, so if the notes cannot show what was asked, what was said, and why a score followed, the score is not defensible, however fair the room felt.