Kafka in Data Pipelines

At-most-once may lose, at-least-once may duplicate, and exactly-once applies each message once via idempotent producers, transactional writes, and offsets committed atomically with the output.

Lag is unprocessed backlog; rebalancing redistributes partitions; back-pressure slows producers; and you stay healthy by scaling consumers, monitoring lag, and dead-lettering poison records.