---
name: "log_forensics"
description: "Diagnose failures from logs and traces: reproduce first, read stack traces correctly, bisect the change, and capture minimal repros."
---

# Log Forensics

Find the actual cause of a failure instead of guessing at it. Works on
crash logs, test failures, CI output, and "it worked yesterday" mysteries.

## Workflow

1. **Reproduce before you theorize.** Run the failing thing once more
   exactly as it failed. If you can't reproduce it, you don't understand
   it yet — note the exact inputs, environment, and time, and keep
   digging. Never "fix" an unreproduced failure.

2. **Read the trace bottom-up, then top-down.**
   - Bottom-up: the last frame / innermost error is usually the
     *proximate* cause (`KeyError: 'foo'`, `ECONNREFUSED`).
   - Top-down: the first frame is *your* code — that's where the fix
     lives. Library internals in the middle are context, not suspects.
   - Read the full message once. Skimming misses the one line that names
     the file and line number.

3. **Separate the error from the noise.**
   - Ignore deprecation warnings, info-level chatter, and retries that
     eventually succeeded. Find the first line where something actually
     *failed*.
   - Timestamps are evidence: a 30s gap before a timeout, or a failure
     at exactly :00, tells you more than the message does.
   - If the log is huge, search for `Traceback`, `Error`, `FATAL`,
     `panic`, `failed`, `exception` — then read 20 lines *above* the
     hit, not just the hit itself.

4. **Bisect the change.** "It worked yesterday" means something changed:
   code, data, config, dependency, environment. Narrow it:
   ```bash
   git log --oneline -15 -- <suspect-file>
   git bisect start; git bisect bad; git bisect good <last-good-sha>
   ```
   If it's not code, diff the environments: dependency versions, env
   vars, config files, and input data between the working and broken
   runs.

5. **Capture a minimal repro.** Shrink the failing case until removing
   anything else makes it pass. A 10-line repro beats a 500-line log for
   getting help and for verifying the fix. Keep the repro — it becomes
   the regression test.

6. **Fix, then verify with the repro — not with your memory of the code.**
   Re-run the exact failing command. If the log is clean, run the wider
   suite once: fixes that break neighbors aren't fixes.

## Output Contract

A diagnosis report contains:
- The reproduced failure (command + output, or "could not reproduce"
  with what was tried)
- The proximate cause (the actual line/condition that failed)
- The root cause (the change or condition that made it fail)
- The fix applied and the verification run that proves it

## Operating Rules

1. Never edit code to "try something" before reproducing the failure.
2. The first hypothesis is usually wrong. Check the evidence before the
   second one, not after the third.
3. "Flaky" is not a diagnosis. Intermittent failure means the trigger is
   timing, ordering, shared state, or resource exhaustion — find which.
4. Don't silence a log line instead of fixing its cause. Suppressed
   errors become outages.
5. When you fix it, write down what the log line *meant* — the next
   agent (or future you) will hit it again.