---
name: "health_checks"
description: "Add readiness and liveness probes to a Python service: /readyz and /livez endpoint design, dependency checks with timeouts, graceful SIGTERM shutdown, and container/K8s probe wiring (httpGet, initialDelaySeconds). Triggers on: add a health endpoint, readiness vs liveness, health check failing, graceful shutdown, K8s probe config, service won't stop on deploy."
metadata: { "includeInPrompt": true }
---

## Purpose

Give a Python service a correct health surface: separate *readiness* (may this instance take traffic?) from *liveness* (is the process stuck and worth restarting?), so dependency outages never trigger restarts and deploys drain cleanly. Scoped to HTTP endpoints, shutdown handling, and container probe wiring — not metrics or alerting.

## Workflow

1. **Split readiness from liveness into two endpoints.** `/livez` answers "is the process alive?"; `/readyz` answers "can it serve traffic right now?". Verified locally: with the dependency down, `/readyz` returned 503 while `/livez` stayed 200 — the orchestrator stops routing but does not restart the pod.

2. **Keep liveness cheap and self-contained.** It must return 200 whenever the process itself is responsive — no database, queue, or upstream calls. What NOT to do: checking upstream deps in liveness turns an outage elsewhere into a restart storm here, killing pods exactly when you need them stable.

3. **Give every readiness check a hard timeout.** Check only what serving traffic actually needs (DB, cache, required upstream), wrap each check in an explicit timeout, and report per-check results. Verified pattern: `socket.create_connection((host, port), timeout=2)` — a dead dependency failed fast with detail instead of hanging the probe. Return 503 naming what failed; keep total check time under the probe's own timeout.

4. **Use a JSON body with per-check detail.** Verified shape:
   `200 {"status":"ok","checks":{"dep":"tcp 127.0.0.1:9999 ok"}}`
   `503 {"status":"not_ready","checks":{"dep":"tcp 127.0.0.1:9999 failed: Connection refused"}}`
   Prefer 503 (not 500) for "not ready": it reads as "route elsewhere", while 500 reads as a bug in your handler.

5. **Drain on SIGTERM before exiting.** In the signal handler, set a draining flag first so `/readyz` immediately returns 503 and the load balancer stops sending traffic; finish in-flight requests; then exit inside the grace window. Verified: SIGTERM flipped readiness to 503 at once, an in-flight 2-second request completed 200, and the process exited 0 within the grace period.

6. **Wire the container probes.** Standard pattern (YAML syntax validated locally — confirm field names against your cluster version before deploying):
   ```yaml
   livenessProbe:  { httpGet: { path: /livez,  port: 8080 }, periodSeconds: 10 }
   readinessProbe: { httpGet: { path: /readyz, port: 8080 }, initialDelaySeconds: 15, periodSeconds: 5, failureThreshold: 2 }
   terminationGracePeriodSeconds: 30
   ```
   `initialDelaySeconds` must exceed your slowest cold start or slow starters get kill-looped; `terminationGracePeriodSeconds` must exceed your drain + shutdown time.

## Output Contract

After applying this skill, an agent should produce:

- `/livez` and `/readyz` handlers (or one documented `/health` treated strictly as readiness) with a sample 200 and a sample 503 response, including per-check detail.
- A SIGTERM handler that flips readiness to 503, drains in-flight requests, and exits 0 within the configured grace — with the drain behavior demonstrated, not assumed.
- Probe wiring (container or K8s YAML) with `initialDelaySeconds` justified against measured startup time and grace periods consistent with the shutdown handler.

## Operating Rules

- **Liveness never touches the network.** The moment a liveness check depends on something else, an outage elsewhere becomes a restart storm here.
- **Timeout on every readiness check, no exceptions.** A check without a timeout is a future hang; size the probe timeout above your worst-case total check time (or run checks concurrently).
- **503 means "not ready", 500 means "broken".** Reserve 500 for handler bugs; orchestrators treat 503 as "send traffic elsewhere".
- **Flip readiness before draining.** The signal handler sets the draining flag first — anything routing traffic must see 503 before in-flight work starts finishing.
- **initialDelaySeconds >= slowest cold start.** Measure it; guessing low causes restart loops on every deploy, guessing high just delays first traffic.
- **terminationGracePeriodSeconds > drain + shutdown.** If the orchestrator kills you mid-drain, in-flight requests die against a half-dead fleet.
- **Don't over-probe.** `periodSeconds` should be comfortably larger than the cost of one full check round, or probes become load.