rag_pipeline_design v1.0.0

name: "rag-pipeline-design" description: "Design and tune RAG pipelines end to end: chunking strategy, embeddings, hybrid retrieval with RRF, reranking, citation-enforced generation, and evaluation loops. Triggers on building a RAG system, fixing bad retrieval, choosing chunk sizes or embedding models, adding reranking, or setting up RAG evaluation."

RAG Pipeline Design

Purpose

Build a retrieval-augmented generation system where each layer has a defined job and a measurable quality bar. Covers the offline pipeline (ingest, chunk, embed, index), the online pipeline (retrieve, fuse, rerank, generate), and the eval loop that keeps all of it honest.

Default stance: hybrid retrieval (BM25 + dense) fused with RRF is the starting point, not a later optimization. Vector-only retrieval misses exact keyword matches; hybrid consistently outperforms either alone.

Workflow

1. Design the ingestion pipeline (offline)

Load documents -> clean and dedupe -> chunk -> embed -> index. Each chunk gets a stable ID and metadata: source, section, timestamp, tenant_id, and a content hash for idempotent re-ingestion.

• Chunk boundaries must be deterministic: same input, same chunks, same IDs. Non-deterministic chunking misaligns embeddings on reindex.
• Chunking strategy per document type, not one size for everything. See references/chunking-strategies.md.
• Default starting point: recursive separator splitting (paragraphs -> sentences -> words), 512 tokens, 10-20% overlap for prose; 256 tokens for code or narrative text. Chunk size is a hyperparameter, not a constant: confirm it on your eval set.
• The embedding model used at index time MUST be the same one used at query time. Keep the model behind an abstraction layer so it can be swapped; pick by benchmark on your own domain, not leaderboard vibes.

2. Build the retrieval path (online)

Query -> enrich -> dense search + BM25 search (parallel) -> fuse -> rerank -> filter -> generate.

• Retrieve top-20 from each channel. Fuse with Reciprocal Rank Fusion: score(d) = sum over lists of 1 / (k + rank(d)), ranks 1-indexed, k=60 (Cormack, Clarke, Buettcher, SIGIR 2009). Items absent from a list contribute zero. RRF discards raw scores, so BM25's unbounded scale and cosine's 0-1 scale never need calibration.
• Rerank the fused top-20 with a cross-encoder (Cohere Rerank, bge-reranker, or similar) and keep top-5.
• Apply metadata filters (tenant, date range, source) BEFORE fusion when the filter is a hard constraint. Enforce tenant isolation in retrieval, not in the prompt.
• Budget the context: dedupe near-duplicate chunks, cap total context tokens, and leave room for the model to think. Never hand the generator 40 chunks and hope.

Query enrichment (cheap, high leverage): expand abbreviations and synonyms before embedding, e.g. "max dose" -> the enriched form with the drug name and context. A bare query is the worst possible retriever input.

3. Enforce citation discipline in generation

See references/citation-discipline.md. The system prompt requires:

• Every factual claim in the answer must carry a citation marker pointing to a retrieved chunk ID.
• No citation, no claim. If the context does not contain the answer, the system says it does not know. Refusal is a feature.
• Conflicting sources get acknowledged with both cited. Never silently pick one.
• The generator must not use its own parametric knowledge for domain facts. Connecting tissue between cited facts (causal links, summaries, "therefore") is itself generated, so treat it as a claim, not a fact.

Post-generation verification (high stakes): verify citations deterministically in code — each marker must exist in the retrieved chunk set — before trusting the LLM. A second-model claim check (decompose into atomic claims, verify each against context) is the heavier option.

4. Run the eval loop

See references/eval-loop.md. Minimum viable eval:

• Keep a fixed set of golden questions with known-good answers.
• Measure per release: Faithfulness (supported claims / total claims in answer), Context Precision (relevant retrieved chunks / total retrieved), Context Recall (ground-truth facts found), Answer Relevancy (answer actually answers the question).
• Re-score the golden set every time the index, chunking, embeddings, or model changes. No re-score, no deploy.
• Tune top-K and the rerank cutoff on eval data, not on gut feel.
• Monitor retrieval and generation independently in production: log which chunks were retrieved per query, the similarity score distribution, and which chunks got cited vs ignored.

5. Plan freshness

• Reindex on a schedule that matches the domain: daily for docs, hourly for real-time content.
• Re-embed the whole corpus when the embedding model changes.
• Track embedding drift: if query-to-chunk similarity distributions shift over months, the index is going stale.

Output Contract

A finished RAG design produces:

1. ChunkConfig for each document type: strategy, size, overlap, separators, metadata schema.
2. Retrieval architecture: channel list, fusion method and parameters (RRF k), candidate counts per stage, reranker choice, metadata filter rules.
3. Grounding contract: the system prompt citation rules, refusal behavior, conflict handling, and the verification step.
4. Eval plan: golden question set, metrics and thresholds, re-score triggers, monitoring dashboards.
5. Freshness plan: reindex cadence, re-embed triggers, drift checks.

Operating Rules

1. When retrieval quality is bad, fix chunking and embeddings first, then retrieval, then prompts. Prompt engineering moves quality last, not first.
2. Never invent benchmark numbers, recall figures, or "typical improvement" claims for a design. Report measured numbers from the eval set or state that a value is an unvalidated starting point.
3. Every chunk ID cited in a generated answer must resolve to a chunk actually retrieved for that query. Markers that point elsewhere get stripped or the answer gets rejected.
4. Chunk size, overlap, top-K, and RRF k are hyperparameters. Write them into a config file, version the config, and re-evaluate after every change.
5. Dense and sparse channels run in parallel and fuse by rank. Never mix raw scores from different scoring scales in one weighted sum without normalization.
6. Keep the source documents authoritative: the index is a derived artifact. When a source document changes, re-chunk and re-embed from the source, never patch the index.
7. Enforce authorization at retrieval time. A generator prompt that says "don't show the user this chunk" is not access control.