Definition
- Apache Pulsar – a distributed messaging and streaming platform with a layered architecture that separates stateless serving brokers from durable BookKeeper storage nodes.
- Apache BookKeeper – the storage layer used by Pulsar, where data is stored as ledger segments distributed across bookie nodes independently of the serving brokers.
- Kafka broker-owns-storage – Kafka's architecture where each broker stores partition log segments on its own local disk, tightly coupling compute and storage.
- Multi-tenancy – Pulsar's native tenant/namespace/topic hierarchy that isolates business units with their own policies, quotas, and access controls on a shared cluster.
- Subscription types – Pulsar's native support for Exclusive, Failover, Shared, and Key_Shared subscriptions, covering both streaming and queue consumption patterns.
- Geo-replication – Pulsar's built-in cross-cluster replication configured per namespace, compared to Kafka's separate MirrorMaker 2 tool.
Imagine two ways to run the vaults of National Bank of India. In the Kafka model, each branch manager keeps the gold in a safe inside their own office: if that office is overwhelmed with customers, moving the gold to another office is slow. In the Pulsar model, the branch managers (brokers) only serve customers at the counter, while the actual gold sits in a shared central vault building (BookKeeper) nearby. If one manager gets busy, another manager instantly serves the same customers because none of them personally hold the gold. Pulsar also keeps separate floors for each client company (multi-tenancy) and automatically copies the vault to another city (geo-replication).
Understanding the Concepts
What is Apache Pulsar?
Apache Pulsar is a distributed messaging and streaming platform whose defining feature is a decoupled architecture that separates serving from storage. Pulsar brokers are stateless routers that handle client connections and message routing but store nothing locally. Durable storage is delegated to Apache BookKeeper nodes called bookies, and cluster metadata lives in a separate metadata store such as ZooKeeper or etcd.
This architectural separation means you can add or remove brokers in seconds without any data movement. In a banking context, when National Bank of India experiences a festival trading spike, new Pulsar brokers can be spun up instantly to handle the load without rebalancing any data.
package com.bank.pulsar;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.Schema;
public class PulsarTransferProducer {
public static void main(String[] args) throws Exception {
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.build();
// Multi-tenant topic naming: tenant/namespace/topic
Producer<String> producer = client.newProducer(Schema.STRING)
.topic("persistent://banking-tenant/payments/transfers")
.create();
producer.send("{\"accountId\":\"SAV2001\",\"amount\":50000}");
System.out.println("Published transfer to Pulsar");
producer.close();
client.close();
}
}
Expected output:
Published transfer to Pulsar
Topic: persistent://banking-tenant/payments/transfers
How does Pulsar's architecture differ from Kafka?
The defining difference is the separation of compute and storage. In Kafka, a broker both serves clients and stores partition log segments on its own disk. When you add a broker to scale, Kafka must physically rebalance partition data across the network, which is slow and I/O heavy. In Pulsar, brokers are stateless routers that own no data. Storage is handled by BookKeeper bookies that store message segments (ledgers) spread across many nodes.
| Feature | Kafka | Pulsar |
|---|---|---|
| Broker role | Serve clients + store data | Serve clients only (stateless) |
| Storage | Broker-local disk (coupled) | BookKeeper bookies (decoupled) |
| Scaling brokers | Requires partition reassignment | Instant (no data movement) |
| Scaling storage | Add broker + reassign partitions | Add bookies independently |
| Data unit | Partition log segments | Ledger segments on bookies |
| Metadata store | KRaft (internal) or ZooKeeper | ZooKeeper or etcd |
package com.bank.comparison;
public class ArchitectureComparison {
public static void main(String[] args) {
System.out.println("Kafka Architecture:");
System.out.println(" Broker 1: serves clients + stores partition 0, 1");
System.out.println(" Broker 2: serves clients + stores partition 2, 3");
System.out.println(" Adding Broker 3: must reassign partitions (slow)");
System.out.println();
System.out.println("Pulsar Architecture:");
System.out.println(" Broker 1: serves clients (no data)");
System.out.println(" Broker 2: serves clients (no data)");
System.out.println(" Bookie 1: stores ledger segments");
System.out.println(" Bookie 2: stores ledger segments");
System.out.println(" Adding Broker 3: instant (no data to move)");
}
}
Expected output:
Kafka Architecture:
Broker 1: serves clients + stores partition 0, 1
Broker 2: serves clients + stores partition 2, 3
Adding Broker 3: must reassign partitions (slow)
Pulsar Architecture:
Broker 1: serves clients (no data)
Broker 2: serves clients (no data)
Bookie 1: stores ledger segments
Bookie 2: stores ledger segments
Adding Broker 3: instant (no data to move)
What is Pulsar's multi-tenancy model?
Multi-tenancy is first-class in Pulsar through the tenant/namespace/topic hierarchy. A topic name looks like persistent://banking-tenant/payments/transfers. The tenant is an organization, the namespace is a grouping with its own policies (retention, quotas, access control), and the topic is the stream. This lets one Pulsar cluster safely host many isolated business units. Kafka has no native tenant concept; isolation is usually done with separate clusters or naming conventions plus ACLs.
package com.bank.pulsar;
public class MultiTenancyExample {
public static void main(String[] args) {
System.out.println("Pulsar Multi-Tenancy at National Bank of India:");
System.out.println();
System.out.println("Tenant: retail-banking");
System.out.println(" Namespace: payments (retention: 7 days, quota: 100GB)");
System.out.println(" Topic: persistent://retail-banking/payments/transfers");
System.out.println(" Topic: persistent://retail-banking/payments/bills");
System.out.println();
System.out.println("Tenant: wealth-management");
System.out.println(" Namespace: trading (retention: 30 days, quota: 500GB)");
System.out.println(" Topic: persistent://wealth-management/trading/orders");
System.out.println();
System.out.println("Tenant: fintech-partner");
System.out.println(" Namespace: api (retention: 3 days, quota: 10GB)");
System.out.println(" Topic: persistent://fintech-partner/api/requests");
System.out.println();
System.out.println("All tenants isolated on ONE Pulsar cluster.");
}
}
Expected output:
Pulsar Multi-Tenancy at National Bank of India:
Tenant: retail-banking
Namespace: payments (retention: 7 days, quota: 100GB)
Topic: persistent://retail-banking/payments/transfers
Topic: persistent://retail-banking/payments/bills
Tenant: wealth-management
Namespace: trading (retention: 30 days, quota: 500GB)
Topic: persistent://wealth-management/trading/orders
Tenant: fintech-partner
Namespace: api (retention: 3 days, quota: 10GB)
Topic: persistent://fintech-partner/api/requests
All tenants isolated on ONE Pulsar cluster.
How does Pulsar handle subscription types?
Pulsar natively supports multiple subscription types in the same topic. Exclusive and Failover subscriptions behave like streaming (one active consumer per subscription). Shared and Key_Shared subscriptions behave like a traditional queue where multiple consumers compete for messages. Kafka only offers the consumer-group model, which is essentially a log-based competing consumer within partitions.
| Subscription Type | Behavior | Kafka Equivalent |
|---|---|---|
| Exclusive | One consumer, fails if another joins | Single-partition topic |
| Failover | Active-standby, auto-failover | Consumer group with 1 partition |
| Shared | Round-robin across consumers (queue) | No direct equivalent |
| Key_Shared | Same key to same consumer (ordered) | Consumer group with key partitioning |
package com.bank.pulsar;
import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.SubscriptionType;
public class SharedSubscriptionConsumer {
public static void main(String[] args) throws Exception {
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.build();
// Shared subscription: multiple consumers compete for messages
// like a traditional queue
Consumer<String> consumer = client.newConsumer(Schema.STRING)
.topic("persistent://banking-tenant/payments/transfers")
.subscriptionName("settlement-sub")
.subscriptionType(SubscriptionType.Shared)
.subscribe();
System.out.println("Shared consumer listening for transfers...");
// Multiple instances compete for messages
consumer.close();
client.close();
}
}
Expected output:
Shared consumer listening for transfers...
-> Instance A processes message 1
-> Instance B processes message 2
-> Instance A processes message 3 (round-robin distribution)
What about Pulsar's built-in geo-replication?
Geo-replication is built into Pulsar at the namespace level. You enable replication to another cluster with a configuration change, and Pulsar handles the cross-region copy transparently. Kafka achieves this with the separate MirrorMaker 2 tool, which is powerful but operationally heavier because you deploy and manage an additional replication cluster.
package com.bank.pulsar;
public class GeoReplicationExample {
public static void main(String[] args) {
System.out.println("Pulsar Geo-Replication:");
System.out.println(" Namespace: banking-tenant/payments");
System.out.println(" Replicated clusters: mumbai, chennai");
System.out.println(" Configuration: one admin API call");
System.out.println();
System.out.println("Kafka Geo-Replication:");
System.out.println(" Tool: MirrorMaker 2 (separate deployment)");
System.out.println(" Source: mumbai-cluster / transfer-events");
System.out.println(" Target: chennai-cluster / mumbai.transfer-events");
System.out.println(" Configuration: separate Connect cluster + config");
}
}
Expected output:
Pulsar Geo-Replication:
Namespace: banking-tenant/payments
Replicated clusters: mumbai, chennai
Configuration: one admin API call
Kafka Geo-Replication:
Tool: MirrorMaker 2 (separate deployment)
Source: mumbai-cluster / transfer-events
Target: chennai-cluster / mumbai.transfer-events
Configuration: separate Connect cluster + config
When would you still prefer Kafka over Pulsar?
Kafka has a far more mature ecosystem: Kafka Connect for hundreds of source and sink connectors, Kafka Streams for stream processing, ksqlDB for SQL on streams, Schema Registry for schema evolution, and enormous community and vendor support. If your organization already runs Kafka and has the operational expertise, and you need the highest raw sequential throughput on existing infrastructure, Kafka is the pragmatic choice.
package com.bank.decision;
public class KafkaVsPulsarDecision {
public static void main(String[] args) {
System.out.println("Choose Kafka when:");
System.out.println(" - Organization already runs Kafka");
System.out.println(" - Need Kafka Connect, Streams, ksqlDB");
System.out.println(" - Highest raw sequential throughput");
System.out.println(" - Large community and vendor support");
System.out.println();
System.out.println("Choose Pulsar when:");
System.out.println(" - Need rapid elastic scaling (festival spikes)");
System.out.println(" - Need multi-tenancy (retail + wealth + fintech)");
System.out.println(" - Need built-in geo-replication");
System.out.println(" - Need both stream and queue in one platform");
System.out.println(" - Need built-in retry + DLT topics");
}
}
Expected output:
Choose Kafka when:
- Organization already runs Kafka
- Need Kafka Connect, Streams, ksqlDB
- Highest raw sequential throughput
- Large community and vendor support
Choose Pulsar when:
- Need rapid elastic scaling (festival spikes)
- Need multi-tenancy (retail + wealth + fintech)
- Need built-in geo-replication
- Need both stream and queue in one platform
- Need built-in retry + DLT topics
What are common mistakes when comparing Kafka and Pulsar?
The most common mistake is treating Pulsar as a drop-in replacement for Kafka without considering the operational cost of running BookKeeper. Another mistake is dismissing Pulsar entirely because Kafka is more popular, when Pulsar's architectural advantages may be a better fit.
package com.bank.decision;
// WRONG: Assuming Pulsar is always better because of decoupled storage.
// You now operate TWO systems: Pulsar brokers AND BookKeeper bookies.
// If your team has Kafka expertise and no need for multi-tenancy,
// the migration cost outweighs the architectural benefit.
// CORRECT: Evaluate based on actual requirements.
public class PulsarMistakes {
public static void main(String[] args) {
System.out.println("WRONG: Migrate to Pulsar just because it is newer.");
System.out.println(" Cost: retrain team, operate BookKeeper, lose");
System.out.println(" Kafka Connect/Streams/ksqlDB ecosystem.");
System.out.println();
System.out.println("CORRECT: Migrate to Pulsar when requirements demand it.");
System.out.println(" - Multi-tenancy needed for isolated divisions");
System.out.println(" - Elastic scaling needed for unpredictable spikes");
System.out.println(" - Both queue and stream semantics on same cluster");
}
}
Expected output:
WRONG: Migrate to Pulsar just because it is newer.
Cost: retrain team, operate BookKeeper, lose
Kafka Connect/Streams/ksqlDB ecosystem.
CORRECT: Migrate to Pulsar when requirements demand it.
- Multi-tenancy needed for isolated divisions
- Elastic scaling needed for unpredictable spikes
- Both queue and stream semantics on same cluster
What should a senior engineer know about this comparison?
A senior engineer should understand that Pulsar and Kafka serve overlapping but not identical needs. The decoupled architecture is not free: you trade Kafka's simpler single-system deployment for Pulsar's extra BookKeeper layer. The right answer depends on organizational context, not just technical features. Kafka's tiered storage feature is narrowing the storage flexibility gap, while Pulsar's ecosystem is maturing but still trails Kafka.
package com.bank.senior;
public class SeniorPerspective {
public static void main(String[] args) {
System.out.println("Senior Engineer Assessment:");
System.out.println("1. Architecture: Pulsar decoupled > Kafka coupled");
System.out.println(" But: Kafka tiered storage narrows the gap");
System.out.println();
System.out.println("2. Ecosystem: Kafka >> Pulsar");
System.out.println(" Connect, Streams, ksqlDB, Schema Registry");
System.out.println();
System.out.println("3. Operations: Kafka = 1 system, Pulsar = 2 systems");
System.out.println(" (brokers + bookies + metadata store)");
System.out.println();
System.out.println("4. Throughput: Kafka wins raw sequential throughput");
System.out.println(" Pulsar wins elastic burst scaling");
}
}
Expected output:
Senior Engineer Assessment:
1. Architecture: Pulsar decoupled > Kafka coupled
But: Kafka tiered storage narrows the gap
2. Ecosystem: Kafka >> Pulsar
Connect, Streams, ksqlDB, Schema Registry
3. Operations: Kafka = 1 system, Pulsar = 2 systems
(brokers + bookies + metadata store)
4. Throughput: Kafka wins raw sequential throughput
Pulsar wins elastic burst scaling
Visual Overview
Kafka vs Pulsar Architecture
============================
Kafka (Broker-Owns-Storage):
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Broker 1 │ │ Broker 2 │ │ Broker 3 │
│ ┌─────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │
│ │ Partition 0 │ │ │ │ Partition 2 │ │ │ │ Partition 4 │ │
│ │ Partition 1 │ │ │ │ Partition 3 │ │ │ │ Partition 5 │ │
│ │ (on disk) │ │ │ │ (on disk) │ │ │ │ (on disk) │ │
│ └─────────────┘ │ │ └─────────────┘ │ │ └─────────────┘ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
Adding broker 4 = must move partition data (SLOW)
Pulsar (Decoupled):
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Broker 1 │ │ Broker 2 │ │ Broker 3 │ <- Stateless
│ (no data)│ │ (no data)│ │ (no data)│ (add/remove fast)
└──────────┘ └──────────┘ └──────────┘
│ │ │
└────────────┼────────────┘
▼
┌──────────────────────────────────────┐
│ BookKeeper Storage Layer │
│ ┌──────────┐ ┌──────────┐ ┌──────┐ │
│ │ Bookie 1 │ │ Bookie 2 │ │Bk. 3│ │
│ │ ledgers │ │ ledgers │ │ldgrs │ │
│ └──────────┘ └──────────┘ └──────┘ │
└──────────────────────────────────────┘
Adding broker 4 = instant (no data to move)
Key Takeaway
Apache Pulsar and Kafka solve the same problem of distributed messaging and streaming, but with fundamentally different architectures. Pulsar separates stateless brokers from BookKeeper storage, so you can add brokers in seconds without moving any data. Kafka couples compute and storage on each broker, which is simpler to deploy but makes scaling slower because adding a broker requires partition reassignment. For National Bank of India during festival trading spikes, Pulsar's instant broker scaling is a clear advantage.
Pulsar also offers first-class multi-tenancy with its tenant/namespace/topic hierarchy, built-in geo-replication configured per namespace, and native support for both streaming and queue subscription types. Kafka achieves multi-tenancy through naming conventions and ACLs, geo-replication through the separate MirrorMaker 2 tool, and queue-like behavior through consumer groups and partitions.
The honest trade-off is that Kafka has a far more mature ecosystem with Kafka Connect, Kafka Streams, ksqlDB, and Schema Registry, plus enormous community support. Pulsar's power comes at the cost of operating the extra BookKeeper layer. If the organization already runs Kafka and does not need elastic scaling, multi-tenancy, or native queue semantics, staying with Kafka is the pragmatic choice.
Remember one sentence: Pulsar decouples brokers from storage for instant scaling and multi-tenancy, while Kafka wins on ecosystem maturity and raw throughput, and the right choice depends on whether you need architectural flexibility or proven operational simplicity.
---