Producing side: outbox

A service that writes to its database and then produces to Kafka has a dual-write problem: either can fail after the other succeeded. The outbox pattern writes the event into an outbox table in the same DB transaction as the business change; a CDC connector (Debezium with the outbox event router) or a poller publishes it. Exactly-once is achieved by the consumer being idempotent (the outbox may publish twice on a crash). Event sourcing (Kafka as the source of truth) is the other extreme; most systems use Kafka as a log of what happened, with a database as state.

Consuming side: idempotence, retries, DLQ

Consumers must tolerate redelivery: dedup on an event id (store processed ids with a TTL, or make the write itself idempotent: upserts, conditional updates). Failures split into transient (retry with backoff) and permanent (DLQ with the error and context headers, alert, tooling to replay). Blocking retries in the poll loop hurt everything behind the record; non-blocking retries publish to topic.retry.5s / topic.retry.1m topics whose consumers wait before processing, then to the DLQ. Note that this breaks per-key ordering for the retried key; if ordering matters, block and pause instead.

Ordering at scale

Ordering is per key per partition; parallelism is the partition count. When you need more consumer parallelism than partitions, process in parallel per key within a consumer (the parallel-consumer pattern), never in arbitrary threads. Reprocessing (replay from an offset or timestamp) must be designed in: idempotent consumers, and side effects (emails, payments) gated by a dedup store.

Cross-cluster & DR

MirrorMaker 2 (on Connect) replicates topics, configs, ACLs and consumer offsets between clusters with source.topic prefixing and offset translation (checkpoints topic), so a consumer group can fail over to the other cluster near where it left off. Active-passive: replicate one way, fail over clients with a config switch; RPO = replication lag, RTO = client redirect time. Active-active: both clusters produce and replicate to each other (prefixed topics avoid loops); consumers read both local and mirrored topics; conflict resolution is your problem. Stretch clusters (one cluster across AZs with rack awareness) give zero RPO within a region; across regions latency makes them fragile. Cloud-native alternatives (cluster linking, geo-replication in managed services) change the mechanics, not the questions.

Capacity and cost

Compute from real numbers: ingress × RF × retention → disk; ingress × (RF + consumers) → network; partition counts → broker count and file handles. Tiered storage moves retention cost to object storage. The expensive things are cross-AZ replication traffic, over-partitioning, and uncompressed JSON.

The capstone

Design, build and break: an order pipeline with outbox → orders topic (Avro, registry) → Streams enrichment + windowed totals → JDBC sink and a search indexer with retry/DLQ; monitored with lag/URP dashboards; secured with SCRAM + ACLs; mirrored to a second cluster. Then run a game day: kill a broker, fill a disk, ship a bad schema, poison a record, expire a certificate. Write the post-mortems.