Producer: Config, Acks, Batching

Use Kafka Connect for standard, config-only data movement with built-in scaling and offset handling, and write your own producer or consumer only when you need custom logic that no connector provides.

With no backpressure, a permanently slow consumer causes lag to grow forever, latency to climb, and eventually retention to delete unread messages -- so you must fix the throughput imbalance, not wait for Kafka to slow the producer.

Bigger batches plus a little linger plus compression plus several in-flight requests maximize throughput, while acks and in-flight settings keep you safe from loss and reordering.

No key means round-robin for even load but no ordering; a key means hash-of-key for per-key ordering; a custom partitioner lets business rules decide, trading off ordering and balance.

The transactional id plus a monotonically increasing epoch fences zombies -- when a partition heals, the old producer's lower epoch is rejected with ProducerFenced, so its stale writes and commits never land.

Deserialization failures happen before your code runs, so you survive them with a safe deserializer plus a dead letter queue, prevent them with Schema Registry validation, and in Streams control them with a DeserializationExceptionHandler.

Graduate transient failures through delayed retry topics with exponential backoff plus jitter so the main consumer never blocks, and send permanent failures straight to the dead letter queue.

The record is durably retained in the log per the retention policy and waits for any future consumer; consumption never deletes it, unlike a queue.

Batch waits for all the data and runs on a schedule for throughput; stream reacts to each event as it arrives for low latency.

Producer needs servers plus serializers plus acks, the broker needs partitions plus replication plus min.insync.replicas, and the consumer needs servers plus deserializers plus a group id.