event_driven_design v1.0.0

name: event_driven_design description: Designs event-driven systems the way that survives production. Triggers when building, reviewing, or fixing event-driven architectures, Kafka/topic design, event modeling, schema registries, ordering, idempotency, DLQs, sagas, or transactional outboxes.

Event-Driven Design

Purpose

Turn "make it async" into a concrete, safe design: well-modeled events, a schema registry with explicit compatibility rules, ordering via partition keys, idempotent consumers, and DLQs that actually catch poison messages. Covers Kafka, RabbitMQ, SQS/SNS, EventBridge, Pub/Sub, and local event buses.

Workflow

1. Model the events first

Run a lightweight event-storm: list what happens in the domain, one fact per line, named [Subject][PastTenseVerb] (OrderPlaced, PaymentProcessed). Separate facts (past tense) from commands (imperative: PlaceOrder) and policies ("whenever X, then Y"). Events are immutable facts; all state transitions get an event.

Run bin/edd-lint.py on your event list to catch naming violations before they fossilize.

2. Fix the envelope

Every event carries the same envelope: event_id (uuid, the dedup key), event_type (orders.placed.v1), event_time (UTC ISO-8601), source, subject (the entity the event is about), plus metadata: correlation_id, causation_id, trace_id. Version the event type with a suffix.

3. Lock the contract in a schema registry

Register Avro / Protobuf / JSON Schema per subject (topic-value naming), with an explicit compatibility mode per subject. See references/schema-evolution.md for the change matrix and the upgrade-order rule. Breaking schema changes must fail at registration time, never at runtime.

4. Decide ordering honestly

Ordering exists only within a Kafka partition, a Kinesis shard, or a FIFO message group. Design the partition key so related events share one: order_id, user_id, tenant_id. More partitions = more parallelism; max consumer parallelism = partition count. If ordering matters cross-key, it does not exist: make consumers tolerate out-of-order arrival (partial records, reconcile later).

5. Make consumers idempotent

At-least-once delivery is the default; duplicates WILL arrive. Dedup on event_id in a store with a uniqueness constraint: seen before means skip, else process and record. Producer side: enable.idempotence=true, acks=all. End-to-end exactly-once needs transactions: use it only where the cost is justified (payments), not everywhere.

6. Publish reliably, fail loudly

Transactional outbox: write the event to an outbox table in the same DB transaction as the business state; a relay publishes to the broker. Never dual-write. Every subscription gets a DLQ with bounded retries (exponential backoff + jitter), alerting on DLQ depth, and a documented replay path. Commit consumer offsets only after durable processing.

7. Choreography or orchestration

Simple flows (3-4 steps, multiple independent consumers): choreography. Complex multi-step sagas with coordinated rollback: orchestration. Common hybrid: orchestrate inside a bounded context, choreograph between them.

Output Contract

When asked to design or review an event-driven system, return:

1. Event catalog: event name, type string, owning service, schema subject, key consumers, partition key.
2. Contract table: per subject, compatibility mode, current version, allowed changes.
3. Ordering map: what ordering each consumer needs, which partition key provides it, where order is explicitly not guaranteed.
4. Failure plan: retry policy per consumer, DLQ topic/queue names, replay procedure, idempotency mechanism.
5. Anti-pattern check: flag events-shaped-as-commands, granularity mistakes, temporal coupling, shared databases, silent unconsumed topics (verify consumption claims with lag metrics, not silence).

Operating Rules

1. Never publish schema-less raw JSON across bounded contexts. The registry is the contract.
2. Never assume ordering across partitions, shards, or groups. Verify against the key, not intuition.
3. Never assume delivery semantics from a flag. Verify idempotency by delivering the same event twice and asserting one effect.
4. One service owns each subject's schema. Consumers adapt; producers do not coordinate with consumers.
5. Events describe what happened (facts). Commands tell someone to act (intent). Never mix the two in one message.
6. An unconsumed topic looks exactly like a quiet topic: every event must have a named consumer and a lag metric from day one.
7. Poison messages never block the main flow: retry bounded, then DLQ, then alert. Forever-retries are a bug.
8. Style: Anthony's rule applies to every word in the skill and its output: no em dashes, no ChatGPT-speak, plain direct language.