--- name: "prompt_debugger" description: "Debugs a failing LLM prompt systematically instead of tweaking it by feel: freeze every variable, reproduce the failure and measure its rate, shrink it to a minimal case, bisect the prompt by removing one section at a time, make one surgical fix, and lock the fix in with an output contract used as a regression test. Triggers when a specific prompt produces a specific wrong, malformed, or inconsistent output, when a prompt that used to work regressed after an edit or a model change, or when someone asks why a prompt misbehaves on some inputs. For designing a new prompt from scratch use prompt-patterns instead; for version history and rollback use prompt-versioning." --- # Prompt Debugger ## Purpose A prompt fails the way code fails: for a findable reason, in a findable place. This skill treats a bad output as a bug report and works it like one. No rewriting the whole prompt and hoping. No stacking new instructions on top of old ones until nobody knows which line does what. The method in one line: reproduce it, shrink it, bisect it, fix one thing, prove the fix, lock it so it stays fixed. Ships with a helper CLI at `bin/prompt_debug.py` (standard library only) that generates section-ablation variants and checks outputs against deterministic contracts. ## Workflow ### 1. Freeze the crime scene Record the exact setup before changing anything: - The full rendered prompt as the model saw it, not the template with placeholders. Template variables, retrieved chunks, and system text all count. - Model name and version, temperature and other sampling settings, tool definitions if any. - The exact input that failed and the exact output it produced. Changing the model, the settings, and the prompt at the same time makes attribution impossible. One variable at a time from here on. ### 2. Reproduce and measure the rate Run the frozen setup on the failing input at least 5 times at production settings. - Fails 5 of 5: deterministic enough to debug directly. - Fails some of the time: it is a rate problem, not a wording problem alone. Record the rate (for example 2 of 5). Fixes are judged by moving the rate, and a single good run proves nothing. - Never reproduces: the frozen copy is missing something the live system had. Diff the rendered prompt and the variable values against the live trace before doing anything else. ### 3. Classify the failure Name the failure before fixing it. The taxonomy in `references/failure-taxonomy.md` covers the ten common classes: instruction conflict, buried instruction, ambiguous spec, examples teaching the wrong pattern, format drift, context dilution, data read as instructions, missing edge case, overconstraint, and plain sampling variance. Each class has a signature and the fix it usually takes. A fix aimed at the wrong class makes the prompt longer and the bug older. ### 4. Shrink to a minimal case Cut the input down while the failure still happens: shorter document, fewer fields, one conversation turn. Stop when removing anything else makes the failure disappear. The minimal case is the test you will rerun all day, and it is what goes into the regression set at the end. If the failure only happens on the long version, that itself is the clue: suspect context dilution or a buried instruction. ### 5. Bisect the prompt Run `python3 bin/prompt_debug.py variants prompt.md --out variants/` to get the full prompt plus one variant per section with that section removed (sections are markdown headings or XML-style tags). Run every variant on the minimal case, same settings, at least 3 runs each for flaky failures. - The variant that fixes the failure names the guilty section. - No single removal fixes it: the cause is an interaction, usually two sections contradicting each other. Remove pairs from the sections that both speak to the failing behavior. - Every variant fails, including heavily cut ones: the cause is outside the prompt sections, so check the model version, the settings, the examples block as data, and the input itself. Worked bisections are in `references/teardowns.md`. ### 6. Form one hypothesis, make one surgical edit Write the hypothesis in one sentence before editing: "The refund example teaches the model to promise refunds, so it promises one on the escalation case." Then make the smallest edit that tests it: fix the example, move the instruction, delete the contradicting line. Resist bundling a cleanup pass into a bug fix. Bundled edits that work leave you not knowing why, and bundled edits that fail leave you not knowing which one hurt. ### 7. Verify against two sets - The minimal failing case: pass at least 5 of 5 runs (or beat the old failure rate clearly on flaky bugs, measured over at least 20 runs). - A regression set: 10 to 30 saved inputs that cover the prompt's main jobs and edge cases, including every past failure you have fixed. A fix that breaks two old cases to save one new one is not a fix. Freeze both sets as files. Past failures become permanent test cases, never just memories. ### 8. Lock it with an output contract Write the behavior the fix restored as a contract and check outputs against it with `python3 bin/prompt_debug.py check output.txt --contract contract.json`. Contracts support required and forbidden strings, regex patterns, JSON validity and required keys, and word-count bounds. Deterministic rules first; where behavior is a judgment call (tone, correct escalation decision), add a short written rubric with observable criteria and grade against it separately, never a vibe. Example contract and promptfoo-style suite layout are in `references/teardowns.md`. Run the contract checks in CI or before any deploy that touches the prompt, exactly like a code test suite. ## Output Contract Every debugging session ends with a short report plus artifacts: - **Repro**: frozen prompt version, model and settings, minimal failing input, original failure rate. - **Classification**: which failure class, and the evidence for it. - **Bisect result**: which section (or pair) caused it, naming the variants run. - **Fix**: the hypothesis and the exact edit, as a diff or before/after snippet. - **Verification**: minimal case result after the fix, regression set result (passes out of total), and the contract file path left behind. ## Operating Rules 1. Never debug against a paraphrase. Work from the rendered prompt and the real output, or say plainly that they are unavailable and the diagnosis is provisional. 2. Never ship a fix verified on one run of a flaky failure. Rates need counts. 3. Never rewrite the whole prompt to fix one failure. A rewrite restarts the bug count at unknown. 4. Deleting an instruction is a legitimate fix. Prompts grow scar tissue; a shorter prompt that passes beats a longer one that argues with itself. 5. When the bisect points outside the prompt (model change, settings, retrieval, tool output), say so and hand the bug to that layer instead of wording around it. 6. Keep the regression set and contracts in the repo next to the prompt. A prompt without its tests will regress silently on the next edit or model swap. 7. Vendor tips and folklore lose to your own measured results on your own cases. Test a trick before adopting it, on the failing case and the regression set.