Kafka Fundamentals: Topics, Partitions, Offsets

Skip Kafka for simple request-reply, for low-volume or tiny-team workloads where a managed queue is cheaper, for priority or complex routing that a classic broker does natively, and when a database transaction is actually the right tool.

Solace routes messages at the broker using a hierarchical topic structure and level-aware wildcards, while Kafka uses flat topics and pushes fine-grained filtering to the consumer.

In Solace the broker is smart and routes each message by its hierarchical topic pattern; in Kafka the broker is simple and the consumer must do the filtering itself.

In Kafka, routing lives in the number of topics so routing dimensions multiply topics combinatorially; in Solace, routing lives in subscription patterns so dimensions cost nothing structurally.

Choose Kafka when you must store and replay the event history; choose Solace when you must route live events flexibly to ever-changing consumers.

Solace routes live events flexibly to the right consumers now; Kafka durably stores every event for replay later; and bridging them gives you both.

Multi-tenant noisy-neighbor is prevented by layered isolation: separate clusters for critical workloads, per-team topics, per-client quotas, partition and broker isolation, per-topic retention, and independent consumer groups.

gzip squeezes hardest but burns CPU, lz4 and snappy are fast but looser, zstd is the balanced modern default, and compression pays off across network and disk because the batch stays compressed until the consumer opens it.

Partitions are the parallelism dial sized by throughput with headroom, bounded by broker and latency limits, and you can only add them, never remove them, because removal would break key ordering and strand data.

Kafka is the durable, replayable event backbone that powers streaming, logging, microservices, pipelines, CDC, and replication from a single stream of truth.

A topic is the category, a partition is one ordered log within that category, and an offset is the exact position of a record inside that partition.

Partitions let Kafka process many records in parallel and scale across machines, while still keeping strict order for all records that share the same key.

Give all instances the same consumer group id so each message is processed by exactly one instance.

Encrypt in transit with TLS, authenticate with mTLS or SASL, authorize with ACLs for least privilege, encrypt at rest, segment the network, and log everything for audit.

A full broker disk takes its partitions offline for writes and stalls replication, so you prevent it with retention, monitoring, quotas, and capacity, and recover by freeing space or adding disk.

Adding partitions changes the modulo divisor in `hash(key) mod numPartitions`, so existing keys get remapped and per-key ordering silently breaks across the split between old and new partitions.

Tiered storage keeps hot recent segments on local disk and offloads cold segments to cheap remote object storage, so Kafka retains data for years cheaply and brokers scale and recover fast.

Kafka is fast on cheap disks because it writes sequentially to an append-only log, serves reads with zero-copy sendfile straight from page cache to socket, and leans on the OS page cache instead of an in-process cache.