---
name: ci_pipeline_design
description: Design CI pipelines that stay fast and trustworthy. Use when building, reviewing, or fixing a CI workflow: caching, parallel jobs, flake quarantine, and keeping full runs under the ten-minute budget.
metadata: { "includeInPrompt": true }
---

# CI Pipeline Design

## Purpose

Build or repair a CI pipeline whose full green run fits inside ten minutes and whose failures always mean a real defect. Slow or flaky CI is one of the highest-leverage engineering problems to fix: DORA's Accelerate research (2018) found pipelines past ten minutes measurably cut developer flow, and unreliable tests destroy confidence in every result the pipeline produces.

## Workflow

1. **Audit the current run.** Find the wall-clock breakdown per stage. The common killers: dependency installs repeated in every job, sequential stages that could run in parallel, superseded PR runs nobody cancels, and a test suite that grows without culling.
2. **Set the budget.** Full green run under ten minutes. Split the budget across stages (e.g. install 1, lint/typecheck 1, unit tests 3, integration/e2e 5) and design every choice against it.
3. **Order stages cheapest first.** Fail fast: lint and unit tests run before integration tests, E2E, and security scans. The pipeline goes red early on cheap checks so it never burns ten minutes to report a lint error.
4. **Parallelize so wall time equals the slowest job, not the sum.** Lint, typecheck, unit, and integration test suites run as parallel jobs. Use matrices only where the matrix earns its keep: libraries tested across versions, apps tested only against the version they deploy. Group short checks together by cost (one job paying runner boot, checkout, and install once), not by concern.
5. **Cache everything deterministic, keyed on the lockfile.** Dependency installs and Docker layers are the cheapest big win. Key every cache off the dependency manifest hash (package-lock.json, go.sum, requirements.txt, Cargo.lock) plus OS/arch, and provide a restore key so a partial hit still helps. For Docker, order the Dockerfile so immutable layers (OS packages, dependencies) come before source, and copy lockfiles before install so source edits do not bust the dependency layer.
6. **Cancel superseded runs.** One config line (`cancel-in-progress` on the ref-based concurrency group) stops old PR pushes from burning minutes on code nobody will ever merge.
7. **Quarantine flakes, never hide them.** When a test fails intermittently: count it, name an owner, move it to a separately-run quarantine suite, and fix it within one sprint. Retry logic is acceptable only if retries are reported, never silently absorbed. Ten reliable tests beat hundreds of untrustworthy ones. Delete tests that cannot be stabilized.
8. **Instrument the pipeline.** Track median time to feedback, flaky test rate, and duration trend. Alert when the median crosses the budget. Review the numbers in retros.
9. **Build once, promote the same artifact.** Never rebuild for staging and again for production. Tag one SHA-tagged artifact and promote it through environments.

## Output Contract

A pipeline design delivered as one of:
- a complete workflow file (e.g. `.github/workflows/ci.yml`) with caching, parallelism, concurrency, and stage ordering in place; or
- a review document listing each stage, its budget, the cache key strategy, the flake-quarantine policy, and the metrics to track.

Either way it names: the ten-minute total budget, the per-stage breakdown, every cache and its key, which jobs run in parallel, the flake policy, and the three metrics from step 8.

See `references/example-workflow.md` for a complete annotated GitHub Actions example and a per-ecosystem caching table.

## Operating Rules

1. Every factual claim about timings, limits, or platform behavior must come from a primary source. Do not invent numbers. Known verified values: DORA's <10-minute finding (Accelerate, 2018); `actions/cache` 10GB free per repo; cache-hit target above 80%; concurrency `cancel-in-progress: true` cancels superseded runs.
2. Never claim a run "passed" when quarantined tests were skipped or silently retried. Report skipped/quarantined tests explicitly.
3. Do not recommend rewriting production code to make CI fast. Fix the pipeline first: caching, parallelism, culling, quarantining.
4. Prefer the smallest runner that fits the job. Bigger runners are a last resort after caching and parallelism are exhausted.
5. Secrets never appear in workflow files or logs. Pin third-party actions to a SHA or major version, never a floating tag.
6. Path filters or affected detection for monorepos: jobs only run when the paths they cover actually changed.