Definition
Apache Pulsar is a distributed messaging and streaming platform that combines the best of Kafka and RabbitMQ in one system.
- Compute-storage separation – brokers are stateless and serve traffic; Apache BookKeeper provides durable, replicated log storage, allowing independent scaling of serving and storage.
- Subscription types – exclusive (one consumer), failover (hot standby), shared (round-robin work queue with per-message ack), and key-shared (per-key ordering with load sharing).
- Multi-tenancy – a built-in tenant, namespace, and topic hierarchy with per-namespace retention, quotas, backlog limits, and access control.
- Geo-replication – native asynchronous replication across clusters in different regions, without separate tooling.
- Unified model – the same topic can be consumed as an ordered stream or as a shared work queue depending on subscription type.
Imagine National Bank of India wants one platform that acts as a permanent ledger when needed and as a shared work queue when needed, and that safely hosts many independent departments on the same infrastructure. Kafka is a great ledger but its work-queue behavior and tenant isolation need extra effort. RabbitMQ is a great work queue but not a durable replayable ledger. Pulsar is the single platform that tries to be both: the ledger lives in a dedicated vault service (BookKeeper), the clerks at the counter (brokers) are interchangeable because they hold no state, each department gets its own walled area (tenant and namespace), and the same document can be replayed like a log or handed out like a work ticket depending on how you subscribe.
Understanding the Concepts
What is Apache Pulsar?
Apache Pulsar is a distributed messaging and streaming platform that combines a durable, append-only log with traditional message queue semantics in a single system. It was originally developed at Yahoo and became an Apache top-level project.
Pulsar's key architectural innovation is separating compute from storage, and its key feature innovation is supporting multiple subscription types so the same topic can be consumed as a stream or as a work queue.
package com.bank.pulsar;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Producer;
import java.math.BigDecimal;
/**
* Rajesh publishes a transaction to Pulsar.
* The same topic can be consumed as a stream or as a work queue.
*/
public class PulsarTransactionProducer {
public static void main(String[] args) throws Exception {
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://pulsar-broker:6650")
.build();
Producer<byte[]> producer = client.newProducer()
.topic("persistent://nbi/payments/transactions")
.create();
String accountId = "SAV2001";
BigDecimal amount = new BigDecimal("75000.00");
String payload = "{\"account\":\"" + accountId
+ "\",\"amount\":" + amount
+ ",\"type\":\"CREDIT\"}";
producer.newMessage()
.key(accountId)
.value(payload.getBytes())
.send();
System.out.println("Published to Pulsar: persistent://nbi/payments/transactions");
System.out.println("Tenant: nbi, Namespace: payments, Topic: transactions");
producer.close();
client.close();
}
}
Expected output:
Published to Pulsar: persistent://nbi/payments/transactions
Tenant: nbi, Namespace: payments, Topic: transactions
How does Pulsar separate compute from storage?
In Kafka, a broker both serves clients and owns the partition data on its local disk. Adding capacity or recovering from a failed broker involves moving large amounts of partition data.
In Pulsar, the broker is stateless. It serves produce and consume traffic but the durable data lives in BookKeeper, a separate distributed log store made of nodes called bookies. A topic's data is a sequence of ledgers striped and replicated across bookies. Because brokers hold no data, you can add or restart a broker quickly and it simply picks up serving topics. Scaling storage and scaling serving are independent decisions.
package com.bank.pulsar;
public class ComputeStorageSeparation {
public static void main(String[] args) {
System.out.println("=== Kafka: Coupled ===");
System.out.println("Broker = serves clients + stores data on local disk");
System.out.println("Adding a broker = rebalance partitions (moves data)");
System.out.println("Broker failure = wait for data to replicate back");
System.out.println();
System.out.println("=== Pulsar: Separated ===");
System.out.println("Broker = serves clients only (stateless)");
System.out.println("BookKeeper bookies = store data (durable, replicated)");
System.out.println("Adding a broker = instant, no data movement");
System.out.println("Broker failure = another broker picks up, no data loss");
System.out.println();
System.out.println("Trade-off: two systems to operate (brokers + bookies)");
}
}
Expected output:
=== Kafka: Coupled ===
Broker = serves clients + stores data on local disk
Adding a broker = rebalance partitions (moves data)
Broker failure = wait for data to replicate back
=== Pulsar: Separated ===
Broker = serves clients only (stateless)
BookKeeper bookies = store data (durable, replicated)
Adding a broker = instant, no data movement
Broker failure = another broker picks up, no data loss
Trade-off: two systems to operate (brokers + bookies)
How do Pulsar subscription types work?
Pulsar supports four subscription types on the same topic. The type determines how messages are delivered.
- Exclusive means one consumer only. If another tries to subscribe, it fails. This gives a single ordered stream, like a Kafka consumer with one partition.
- Failover means one active consumer with hot standbys. If the active consumer disconnects, a standby takes over.
- Shared means messages are distributed round-robin across consumers with per-message acknowledgement. This is a classic work queue where each message goes to one worker, and the worker acks individually.
- Key-shared means messages with the same key always go to the same consumer, preserving per-key order while sharing load across consumers.
package com.bank.pulsar;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.SubscriptionType;
import org.apache.pulsar.client.api.Message;
import java.util.concurrent.TimeUnit;
/**
* Settlement workers use Shared subscription at National Bank of India.
* Each trade is processed by exactly one worker, with per-message ack.
*/
public class SharedSubscriptionWorker {
public static void main(String[] args) throws Exception {
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://pulsar-broker:6650")
.build();
Consumer<byte[]> consumer = client.newConsumer()
.topic("persistent://nbi/payments/transactions")
.subscriptionName("settlement-workers")
.subscriptionType(SubscriptionType.Shared) // work queue!
.subscribe();
System.out.println("Worker started with Shared subscription.");
System.out.println("Messages distributed round-robin across workers.");
Message<byte[]> msg = consumer.receive(5, TimeUnit.SECONDS);
if (msg != null) {
String payload = new String(msg.getValue());
System.out.println("Processing trade: " + payload);
// Per-message ack: this specific message is done
consumer.acknowledge(msg);
System.out.println("Acked individually. Other workers process other trades.");
}
consumer.close();
client.close();
}
}
Expected output:
Worker started with Shared subscription.
Messages distributed round-robin across workers.
Processing trade: {"account":"SAV2001","amount":75000.00,"type":"CREDIT"}
Acked individually. Other workers process other trades.
How does Pulsar multi-tenancy work?
Pulsar has a built-in hierarchy: tenant, then namespace, then topic. A tenant is an isolated organizational unit. A namespace groups topics and carries policies such as retention, quotas, backlog limits, and access control.
This lets many teams share one Pulsar cluster with real isolation. In Kafka, multi-tenancy is achieved with naming conventions, ACLs, and quotas, which is capable but not a built-in model.
package com.bank.pulsar;
public class MultiTenancyDemo {
public static void main(String[] args) {
System.out.println("Pulsar multi-tenancy at National Bank of India:");
System.out.println();
System.out.println("Tenant: retail-banking");
System.out.println(" Namespace: payments");
System.out.println(" Topic: persistent://retail-banking/payments/transactions");
System.out.println(" Retention: 30 days, Quota: 100 GB");
System.out.println(" Namespace: loans");
System.out.println(" Topic: persistent://retail-banking/loans/applications");
System.out.println(" Retention: 90 days, Quota: 50 GB");
System.out.println();
System.out.println("Tenant: trading");
System.out.println(" Namespace: equities");
System.out.println(" Topic: persistent://trading/equities/orders");
System.out.println(" Retention: 7 years, Quota: 500 GB");
System.out.println();
System.out.println("Each tenant is isolated with its own policies.");
System.out.println("One cluster serves both retail and trading.");
}
}
Expected output:
Pulsar multi-tenancy at National Bank of India:
Tenant: retail-banking
Namespace: payments
Topic: persistent://retail-banking/payments/transactions
Retention: 30 days, Quota: 100 GB
Namespace: loans
Topic: persistent://retail-banking/loans/applications
Retention: 90 days, Quota: 50 GB
Tenant: trading
Namespace: equities
Topic: persistent://trading/equities/orders
Retention: 7 years, Quota: 500 GB
Each tenant is isolated with its own policies.
One cluster serves both retail and trading.
How does Pulsar compare to Kafka?
| Feature | Kafka | Pulsar |
|---|---|---|
| Architecture | Broker stores data locally | Stateless brokers + BookKeeper storage |
| Scaling | Scale broker = move partitions | Scale brokers and bookies independently |
| Subscription types | Consumer groups only | Exclusive, failover, shared, key-shared |
| Per-message ack | No (offset-based) | Yes (shared and key-shared subscriptions) |
| Multi-tenancy | Naming conventions + ACLs + quotas | Built-in tenant/namespace hierarchy |
| Geo-replication | MirrorMaker 2 or third-party | Built-in native replication |
| Stream processing | Kafka Streams, ksqlDB | Pulsar Functions (lighter weight) |
| Ecosystem maturity | Very mature, broad tooling | Growing, fewer connectors and clients |
| Operational complexity | One system (brokers + ZK/KRaft) | Two systems (brokers + BookKeeper + metadata store) |
package com.bank.comparison;
public class KafkaVsPulsar {
public static void main(String[] args) {
System.out.println("Kafka strengths:");
System.out.println(" - Mature ecosystem (Streams, ksqlDB, Connect)");
System.out.println(" - Broad client and connector support");
System.out.println(" - Widely available operational expertise");
System.out.println(" - Simpler architecture (one system)");
System.out.println();
System.out.println("Pulsar strengths:");
System.out.println(" - Compute-storage separation");
System.out.println(" - Shared subscriptions (work queue + streaming)");
System.out.println(" - Built-in multi-tenancy");
System.out.println(" - Native geo-replication");
System.out.println();
System.out.println("Trade-off: Pulsar's elegance vs Kafka's ecosystem maturity");
}
}
Expected output:
Kafka strengths:
- Mature ecosystem (Streams, ksqlDB, Connect)
- Broad client and connector support
- Widely available operational expertise
- Simpler architecture (one system)
Pulsar strengths:
- Compute-storage separation
- Shared subscriptions (work queue + streaming)
- Built-in multi-tenancy
- Native geo-replication
Trade-off: Pulsar's elegance vs Kafka's ecosystem maturity
When should you consider Pulsar over Kafka?
Consider Pulsar when specific features outweigh Kafka's ecosystem advantage.
- Multi-tenancy is the primary requirement. Many independent teams or clients share one platform and need real isolation with per-namespace policies.
- Unified queue plus stream in one system. You currently run both Kafka and RabbitMQ and want to consolidate.
- Built-in geo-replication across regions without operating MirrorMaker 2 separately.
- Independent scaling of compute and storage. Stateless brokers make scaling serving trivially fast.
package com.bank.comparison;
public class WhenToConsiderPulsar {
public static void main(String[] args) {
System.out.println("Consider Pulsar when:");
System.out.println(" 1. Multi-tenancy is primary (tenant/namespace isolation)");
System.out.println(" 2. Need both queue + stream from same topic");
System.out.println(" 3. Want built-in geo-replication (not MirrorMaker)");
System.out.println(" 4. Want independent compute/storage scaling");
System.out.println();
System.out.println("Stick with Kafka when:");
System.out.println(" 1. Ecosystem maturity matters (Streams, ksqlDB, Connect)");
System.out.println(" 2. Operational expertise is easier to hire");
System.out.println(" 3. Fewer moving parts preferred (no BookKeeper)");
System.out.println(" 4. Existing Kafka investment is significant");
}
}
Expected output:
Consider Pulsar when:
1. Multi-tenancy is primary (tenant/namespace isolation)
2. Need both queue + stream from same topic
3. Want built-in geo-replication (not MirrorMaker)
4. Want independent compute/storage scaling
Stick with Kafka when:
1. Ecosystem maturity matters (Streams, ksqlDB, Connect)
2. Operational expertise is easier to hire
3. Fewer moving parts preferred (no BookKeeper)
4. Existing Kafka investment is significant
What are common mistakes?
Wrong: choosing Pulsar only because the architecture is "better."
Pulsar's architecture is elegant but has more moving parts. You run brokers, bookies, and a metadata store. Kafka's KRaft mode is simpler: just brokers.
package com.bank.comparison;
public class WrongPulsarForArchitecture {
public static void main(String[] args) {
System.out.println("WRONG: Choosing Pulsar because 'separation is cleaner'");
System.out.println(" Pulsar components: brokers + bookies + metadata store");
System.out.println(" More processes to deploy, monitor, and tune");
System.out.println(" Fewer operators know Pulsar than Kafka");
System.out.println();
System.out.println("CORRECT: Choose Pulsar for specific capabilities");
System.out.println(" Multi-tenancy, shared subscriptions, geo-replication");
System.out.println(" Not just because the architecture diagram looks elegant");
}
}
Expected output:
WRONG: Choosing Pulsar because 'separation is cleaner'
Pulsar components: brokers + bookies + metadata store
More processes to deploy, monitor, and tune
Fewer operators know Pulsar than Kafka
CORRECT: Choose Pulsar for specific capabilities
Multi-tenancy, shared subscriptions, geo-replication
Not just because the architecture diagram looks elegant
What about Pulsar's geo-replication?
Pulsar has built-in geo-replication. You configure a namespace to replicate across clusters in different regions, and Pulsar handles the asynchronous cross-region replication natively. In Kafka, this requires MirrorMaker 2 or a commercial replicator, which is an additional component to operate.
package com.bank.pulsar;
public class GeoReplicationDemo {
public static void main(String[] args) {
System.out.println("Pulsar geo-replication:");
System.out.println(" Namespace: nbi/payments");
System.out.println(" Clusters: mumbai-cluster, chennai-cluster");
System.out.println(" Config: set replication clusters on namespace");
System.out.println(" Result: messages automatically replicated between regions");
System.out.println();
System.out.println("Kafka geo-replication:");
System.out.println(" Tool: MirrorMaker 2 (separate Connect cluster)");
System.out.println(" Config: source and target cluster properties");
System.out.println(" Result: works well but is an additional system to operate");
}
}
Expected output:
Pulsar geo-replication:
Namespace: nbi/payments
Clusters: mumbai-cluster, chennai-cluster
Config: set replication clusters on namespace
Result: messages automatically replicated between regions
Kafka geo-replication:
Tool: MirrorMaker 2 (separate Connect cluster)
Config: source and target cluster properties
Result: works well but is an additional system to operate
What should a senior engineer know?
The honest senior assessment balances Pulsar's features against Kafka's maturity.
- Kafka has a far larger ecosystem: Kafka Streams, ksqlDB, hundreds of Connect connectors, Schema Registry, and broad client support. Pulsar's ecosystem is growing but smaller.
- Operational expertise for Kafka is more widely available, which matters for hiring and on-call rotation.
- Pulsar's architecture has more moving parts (brokers, bookies, metadata store), so it can be harder to operate well.
- Pulsar Functions provide lightweight stream processing but are not as powerful as Kafka Streams.
- The decision should weigh Pulsar's native multi-tenancy, unified queue-plus-stream model, and built-in geo-replication against Kafka's ecosystem maturity and operational familiarity.
package com.bank.comparison;
public class SeniorPulsarAssessment {
public static void main(String[] args) {
System.out.println("Senior assessment:");
System.out.println("1. Kafka ecosystem >> Pulsar ecosystem (today)");
System.out.println("2. Kafka operational expertise is easier to hire");
System.out.println("3. Pulsar has more moving parts (brokers + bookies + meta)");
System.out.println("4. Pulsar Functions < Kafka Streams in power");
System.out.println("5. Pulsar multi-tenancy is genuinely better than Kafka's");
System.out.println("6. Pulsar shared subscription is genuinely useful");
System.out.println("7. Weigh features vs ecosystem maturity for your case");
}
}
Expected output:
Senior assessment:
1. Kafka ecosystem >> Pulsar ecosystem (today)
2. Kafka operational expertise is easier to hire
3. Pulsar has more moving parts (brokers + bookies + meta)
4. Pulsar Functions < Kafka Streams in power
5. Pulsar multi-tenancy is genuinely better than Kafka's
6. Pulsar shared subscription is genuinely useful
7. Weigh features vs ecosystem maturity for your case
Visual Overview
Pulsar Architecture:
┌─────────────────────────────────────────────┐
│ Pulsar Cluster │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Broker 1 │ │ Broker 2 │ │ Broker 3 │ │
│ │(stateless)│ │(stateless)│ │(stateless)│ │
│ └─────┬────┘ └─────┬────┘ └─────┬────┘ │
│ │ │ │ │
│ └─────────────┼─────────────┘ │
│ │ │
│ ┌───────────────────▼──────────────────┐ │
│ │ Apache BookKeeper │ │
│ │ ┌────────┐ ┌────────┐ ┌────────┐ │ │
│ │ │Bookie 1│ │Bookie 2│ │Bookie 3│ │ │
│ │ │(data) │ │(data) │ │(data) │ │ │
│ │ └────────┘ └────────┘ └────────┘ │ │
│ └──────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────┐ │
│ │ Metadata Store (ZK/etcd) │ │
│ └──────────────────────────────────────┘ │
└─────────────────────────────────────────────┘
Subscription Types:
Topic: persistent://nbi/payments/transactions
│
├── Exclusive: [Consumer A] only
├── Failover: [Consumer B active] [C standby]
├── Shared: [Worker 1] [Worker 2] [Worker 3]
│ round-robin, per-message ack
└── Key-shared: [W1: key=SAV2001] [W2: key=CUR3001]
per-key order preserved
Key Takeaway
Apache Pulsar is a distributed messaging and streaming platform that combines Kafka-like durable log storage with RabbitMQ-like queue semantics in one system. Its most distinctive choice is separating compute from storage: brokers are stateless and serve traffic, while BookKeeper bookies provide the durable replicated log. This means you can scale serving and storage independently, and a broker restart involves no data movement.
What makes Pulsar unified is its subscription types. The same topic can be consumed as an exclusive ordered stream (like Kafka), as a shared round-robin work queue with per-message acknowledgement (like RabbitMQ), or as a key-shared subscription that preserves per-key order while distributing load. Add built-in multi-tenancy with tenant and namespace hierarchies and native geo-replication, and Pulsar addresses several gaps that require separate tools in a Kafka deployment.
At National Bank of India, the evaluation was concrete. The settlement team needed per-message ack on a worker pool, which Pulsar's shared subscription provides natively. The tenant/namespace model let retail payments and trading share one cluster with isolated policies. The built-in geo-replication covered the Mumbai-to-DR requirement without operating MirrorMaker. The honest trade-off is that Kafka has a far larger ecosystem, more mature tooling, and more available expertise.
Remember one sentence: Pulsar gives you a queue and a log from the same topic, with built-in multi-tenancy and geo-replication, but weigh those features against Kafka's larger ecosystem and simpler operational model.
---