health_checks v1.0.0

name: "health_checks" description: "Add readiness and liveness probes to a Python service: /readyz and /livez endpoint design, dependency checks with timeouts, graceful SIGTERM shutdown, and container/K8s probe wiring (httpGet, initialDelaySeconds). Triggers on: add a health endpoint, readiness vs liveness, health check failing, graceful shutdown, K8s probe config, service won't stop on deploy." metadata: { "includeInPrompt": true }

Purpose

Give a Python service a correct health surface: separate readiness (may this instance take traffic?) from liveness (is the process stuck and worth restarting?), so dependency outages never trigger restarts and deploys drain cleanly. Scoped to HTTP endpoints, shutdown handling, and container probe wiring — not metrics or alerting.

Workflow

1. Split readiness from liveness into two endpoints. /livez answers "is the process alive?"; /readyz answers "can it serve traffic right now?". Verified locally: with the dependency down, /readyz returned 503 while /livez stayed 200 — the orchestrator stops routing but does not restart the pod.

1. Keep liveness cheap and self-contained. It must return 200 whenever the process itself is responsive — no database, queue, or upstream calls. What NOT to do: checking upstream deps in liveness turns an outage elsewhere into a restart storm here, killing pods exactly when you need them stable.

1. Give every readiness check a hard timeout. Check only what serving traffic actually needs (DB, cache, required upstream), wrap each check in an explicit timeout, and report per-check results. Verified pattern: socket.create_connection((host, port), timeout=2) — a dead dependency failed fast with detail instead of hanging the probe. Return 503 naming what failed; keep total check time under the probe's own timeout.

1. Use a JSON body with per-check detail. Verified shape: 200 {"status":"ok","checks":{"dep":"tcp 127.0.0.1:9999 ok"}} 503 {"status":"not_ready","checks":{"dep":"tcp 127.0.0.1:9999 failed: Connection refused"}} Prefer 503 (not 500) for "not ready": it reads as "route elsewhere", while 500 reads as a bug in your handler.

1. Drain on SIGTERM before exiting. In the signal handler, set a draining flag first so /readyz immediately returns 503 and the load balancer stops sending traffic; finish in-flight requests; then exit inside the grace window. Verified: SIGTERM flipped readiness to 503 at once, an in-flight 2-second request completed 200, and the process exited 0 within the grace period.

1. Wire the container probes. Standard pattern (YAML syntax validated locally — confirm field names against your cluster version before deploying):

yaml
   livenessProbe:  { httpGet: { path: /livez,  port: 8080 }, periodSeconds: 10 }
   readinessProbe: { httpGet: { path: /readyz, port: 8080 }, initialDelaySeconds: 15, periodSeconds: 5, failureThreshold: 2 }
   terminationGracePeriodSeconds: 30
   

initialDelaySeconds must exceed your slowest cold start or slow starters get kill-looped; terminationGracePeriodSeconds must exceed your drain + shutdown time.

Output Contract

After applying this skill, an agent should produce:

• /livez and /readyz handlers (or one documented /health treated strictly as readiness) with a sample 200 and a sample 503 response, including per-check detail.
• A SIGTERM handler that flips readiness to 503, drains in-flight requests, and exits 0 within the configured grace — with the drain behavior demonstrated, not assumed.
• Probe wiring (container or K8s YAML) with initialDelaySeconds justified against measured startup time and grace periods consistent with the shutdown handler.

Operating Rules

• Liveness never touches the network. The moment a liveness check depends on something else, an outage elsewhere becomes a restart storm here.
• Timeout on every readiness check, no exceptions. A check without a timeout is a future hang; size the probe timeout above your worst-case total check time (or run checks concurrently).
• 503 means "not ready", 500 means "broken". Reserve 500 for handler bugs; orchestrators treat 503 as "send traffic elsewhere".
• Flip readiness before draining. The signal handler sets the draining flag first — anything routing traffic must see 503 before in-flight work starts finishing.
• initialDelaySeconds >= slowest cold start. Measure it; guessing low causes restart loops on every deploy, guessing high just delays first traffic.
• terminationGracePeriodSeconds > drain + shutdown. If the orchestrator kills you mid-drain, in-flight requests die against a half-dead fleet.
• Don't over-probe. periodSeconds should be comfortably larger than the cost of one full check round, or probes become load.