decision_journals v1.0.0

name: "decision_journals" description: "Keep a personal decision journal: write down each decision with its context and a predicted outcome plus a confidence number BEFORE the result is known, then review on a schedule to score predictions against outcomes and calibrate judgment. Use when someone wants to learn from their past decisions, track predictions against what actually happened, or check whether their confidence matches reality." metadata: { "includeInPrompt": true }

Decision Journals

Purpose

Build a personal learning loop around decisions. Before you act, you write down what you expect to happen and how confident you are. Later, after the result is in, you compare the two. Over time this shows where your judgment is sharp and where you are consistently overconfident, so future calls get better.

This is not an architecture decision record. decision_records (ADRs) is an immutable team record of why a technical choice was made, for future contributors to read. A decision journal is personal and predictive: its value is the gap between what you wrote down before and what happened after. It is also not general note-taking. note_taking_systems covers capture methods like Cornell and Zettelkasten. A journal covers one narrow thing: predictions and outcomes, scored.

Workflow

1. Pick what to journal. When a decision is hard to reverse, costly, or a repeat of a past call, it earns an entry. When it is cheap and reversible, skip it. Journaling everything turns the habit into homework and the habit dies.

1. Write the entry before acting. Use the template below. The prediction is the load-bearing part: it must name an observable outcome inside a time window ("revenue up 20% in 60 days"), plus a confidence number from 0 to 100. Ask the forcing questions in section "Prediction prompts" until you get a real number and a real observable, not "probably fine."

1. Lock it. Once the entry is written, do not edit the prediction. If new information changes the picture, that is a new decision with a new entry. Editing predictions after the fact destroys the whole point.

1. Set a review date. Pick the date when the outcome will actually be observable, not sooner. Guessing early produces noise.

1. Run the review pass on a cadence. Weekly or monthly, open every entry past its review date. Write down the actual outcome as 1 (happened), 0 (did not happen), or a fraction like 0.5 for a partial. Then score with the calibration script below.

1. Read the bins. If your 80-90% predictions hit only 40% of the time in some domain, say so plainly and state a lower confidence range for that domain next time. If a bin is well calibrated, trust the gut there.

1. Write one adjustment. End each review with one concrete change for next time ("in hiring calls, cap my stated confidence at 60% until the hit rate earns higher"). Keep it to one. A list of twelve gets ignored.

Entry template

Save one file per decision, dated, somewhere simple like ~/decisions/2026-09-27-switch-suppliers.md:

markdown
# Decision: <short title>
Date: 2026-09-27
Review by: 2026-11-27

## Context
(What was going on, in a few lines. Only what mattered to the call.)

## Decision
(What you decided, in one or two sentences.)

## Options considered
1. <option> - <why it won or lost>
2. <option> - <why it won or lost>

## Prediction
<I expect to see...> (observable outcome, with a time window)
Confidence: 70%

## What would prove me wrong
(Specific and observable. Write it now, not after you know.)

## Outcome (fill in at review)
Happened / did not happen / partial. Score: 1 / 0 / 0.5
Notes:

Prediction prompts

Use these to force a prediction without hindsight bias. Stop when the answer is a number plus an observable, not a feeling.

• "If you had to bet your own money, which side, and how much?"
• "Put a number on it. 60%? 90%? A number, not 'probably'."
• "What will you see by <date> if you are right? What will you see if you are wrong?"
• "What did you choose not to do, and why?"
• "What would a smart skeptic say you are missing?"
• "When will you actually know the answer?" (This sets the review date.)

Calibration script

Save this as ~/decisions/calibrate.py. It reads a JSON list of reviewed entries and reports a Brier-style score (mean of (confidence minus outcome) squared, lower is better) against an always-50% baseline, plus confidence bins comparing stated confidence to actual hit rate. Stdlib only. Verified locally against a 10-entry fixture, including the empty and all-unresolved edge cases.

Run it like this:

mkdir -p ~/decisions
python3 ~/decisions/calibrate.py ~/decisions/reviewed.json

reviewed.json is a list of entries with outcomes filled in:

json
[
  {
    "date": "2026-08-15",
    "decision": "Launch feature X early",
    "prediction": "No major bugs in week 1",
    "confidence": 0.9,
    "outcome": 0
  }
]

• confidence: 0.0 to 1.0, written before the outcome.
• outcome: 1 (happened), 0 (did not happen), a fraction for partial, or null while still open. Null entries are skipped by the scorer.

python
#!/usr/bin/env python3
"""calibrate.py - score decision-journal predictions against outcomes.

Reads a JSON file of reviewed journal entries and reports a Brier-style
calibration score plus a confidence-binned breakdown.

Reviewed-entry JSON format (a list of objects):

  [
    {
      "date": "2026-09-20",
      "decision": "Switch suppliers to Acme",
      "prediction": "Costs drop 10% by Q4",
      "confidence": 0.75,
      "outcome": 1
    }
  ]

- confidence: your stated probability, 0.0 to 1.0, written BEFORE the outcome.
- outcome: 1 = it happened, 0 = it did not, a fraction like 0.5 for
  a partial, null = still open (skipped by the scorer).

Usage:
  python3 calibrate.py reviewed.json
"""

import json
import sys


def main():
    if len(sys.argv) != 2:
        print("usage: python3 calibrate.py reviewed.json")
        sys.exit(1)
    with open(sys.argv[1]) as f:
        entries = json.load(f)
    if not isinstance(entries, list):
        print("reviewed.json must hold a JSON list of entries.")
        sys.exit(1)

    scored = [e for e in entries
              if isinstance(e.get("outcome"), (int, float))]
    waiting = len(entries) - len(scored)  # only null-outcome entries are waiting
    if not scored:
        print("No resolved entries to score yet.")
        sys.exit(0)

    # Brier-style score: mean of (confidence - outcome)^2. Lower is better.
    brier = sum((e["confidence"] - e["outcome"]) ** 2 for e in scored) / len(scored)
    # Baseline: a person who says 50% on everything.
    baseline = sum((0.5 - e["outcome"]) ** 2 for e in scored) / len(scored)

    print(f"Resolved entries: {len(scored)} ({waiting} still open)")
    print(f"Brier score (lower is better): {brier:.3f}")
    print(f"Always-50% baseline:           {baseline:.3f}")
    print(f"Beats baseline: {'yes' if brier < baseline else 'not yet'}")

    print("\nCalibration by confidence bin (avg confidence vs actual hit rate):")
    for pct in range(0, 100, 10):
        lo, hi = pct / 100, (pct + 10) / 100
        inbin = [e for e in scored if lo <= e["confidence"] < hi or (hi >= 1.0 and e["confidence"] == 1.0)]
        if inbin:
            avg_c = sum(e["confidence"] for e in inbin) / len(inbin)
            hit = sum(e["outcome"] for e in inbin) / len(inbin)
            flag = ""
            if avg_c - hit > 0.1:
                flag = "  <-- overconfident here"
            elif hit - avg_c > 0.1:
                flag = "  <-- underconfident here"
            print(f"  {pct:2d}-{pct+10:3d}%: n={len(inbin):2d}  avg conf {avg_c:.2f}  hit rate {hit:.2f}{flag}")

    key = lambda e: (e["confidence"] - e["outcome"]) ** 2  # noqa: E731
    best = min(scored, key=key)
    worst = max(scored, key=key)
    print(f"\nBest call:  {best['decision'][:60]} (conf {best['confidence']:.0%}, outcome {best['outcome']})")
    print(f"Worst miss: {worst['decision'][:60]} (conf {worst['confidence']:.0%}, outcome {worst['outcome']})")


if __name__ == "__main__":
    main()

Output Contract

When this skill fires on a decision, it produces:

• A dated entry file following the template: context, decision, options considered, predicted outcome with confidence, what would prove the call wrong, and a review date.
• At review time: outcome scores on resolved entries and a calibration summary from the script (Brier-style score vs the always-50% baseline, per-bin confidence vs hit rate, best and worst calls, and one adjustment for next time).

Done looks like this: every resolved decision has a prediction written before the outcome was known, no prediction was edited after the fact, and the review summary names at least one concrete adjustment for the next round.

Operating Rules

1. When a decision is cheap and reversible, skip journaling it. The habit survives only if entries stay rare and feel worth the effort.
2. When writing the prediction, make the outcome observable and time-bound. "Things go well" cannot be scored, so rewrite it until it can.
3. When the outcome arrives, score the entry as written. Never rewrite the prediction to fit reality; that turns the journal into a diary.
4. When a confidence bin shows stated confidence more than about 10 points above the hit rate, name the overconfidence and recommend a lower range for that domain next time.
5. When fewer than about 10 entries are resolved, treat the numbers as early signal, not a verdict. Calibration settles with volume, and a small sample will bounce around.
6. When the ask is really about a team or a codebase rather than a person learning from their own calls, recommend decision_records (ADRs) instead. That cousin is built for the team job; this one is built for the personal loop.