Definition
Topic proliferation is the operational problem that occurs when Kafka's flat topic model meets multiple routing dimensions.
- Topic proliferation – the explosive growth of Kafka topics when routing dimensions (type, country, city, size) are encoded as separate topics, resulting in a combinatorial product.
- Per-topic cost – each Kafka topic requires explicit creation, partition count, replication factor, retention settings, ACLs, and monitoring, making hundreds of topics operationally expensive.
- Partition ceiling – a Kafka cluster has a practical limit on total partitions because each partition consumes broker memory, file handles, and controller metadata.
- Solace avoidance – Solace uses one hierarchical namespace where topics are publish-time strings and routing is a subscriber concern expressed with wildcards, so no new broker objects are created.
- Change cost – adding a new dimension value (new country, new type) in Kafka means creating new topics and rewiring producers; in Solace it is just a new subscription string.
Imagine National Bank of India wants a separate physical mailbox for every combination of transaction type, branch city, and amount band. With three types, fifty branches, and four amount bands you need six hundred mailboxes, each installed, labelled, locked, and inspected by the facilities team. That is Kafka topic proliferation. Solace instead has one smart sorting room: every letter is addressed hierarchically, and a desk that wants a new slice just tells the sorting clerk a new address pattern. No new mailbox is ever built.
Understanding the Concepts
What is the topic proliferation problem?
Kafka routes at topic granularity. The broker does not look inside a record, so the only way for a consumer to receive just a subset without downloading everything is for that subset to already be its own topic. This forces the routing decision to publish time: the publisher must decide which topic a record belongs to.
The moment you have more than one routing dimension, the topic count becomes a product. If you route transactions by type (credit, debit, transfer), by country (IN, BR, US, UK), and by size (small, medium, large), that is 3 x 4 x 3 = 36 topics. With fifty cities instead of four countries, that jumps to 450.
package com.bank.kafka;
public class TopicProliferationCalculator {
public static void main(String[] args) {
int types = 3; // credit, debit, transfer
int countries = 4; // IN, BR, US, UK
int sizes = 3; // small, medium, large
int topics = types * countries * sizes;
System.out.println("Dimensions: " + types + " types x "
+ countries + " countries x " + sizes + " sizes");
System.out.println("Topics needed: " + topics);
System.out.println();
// With realistic numbers
int cities = 50;
int accountTypes = 4; // savings, current, loan, credit-card
int realTopics = types * cities * accountTypes * sizes;
System.out.println("Real dimensions: " + types + " types x "
+ cities + " cities x " + accountTypes + " acct types x " + sizes + " sizes");
System.out.println("Topics needed: " + realTopics);
System.out.println();
int partitionsPerTopic = 6;
System.out.println("Total partitions: " + (realTopics * partitionsPerTopic));
System.out.println("Kafka recommended max: ~200,000 partitions per cluster");
}
}
Expected output:
Dimensions: 3 types x 4 countries x 3 sizes
Topics needed: 36
Real dimensions: 3 types x 50 cities x 4 acct types x 3 sizes
Topics needed: 1800
Total partitions: 10800
Kafka recommended max: ~200,000 partitions per cluster
Why is each topic expensive?
Each Kafka topic is not free. Every topic needs:
- Explicit creation often gated by a platform team's approval process.
- Partition count chosen for expected throughput.
- Replication factor for durability.
- Retention settings for how long to keep records.
- ACLs controlling who can produce and consume.
- Monitoring for under-replicated partitions, consumer lag, and disk usage.
Partitions consume broker resources: each partition maps to a directory on disk, a set of file handles, and metadata in the controller. A cluster has a practical ceiling, and uncontrolled topic growth can destabilize it.
package com.bank.kafka;
import java.math.BigDecimal;
public class PerTopicCost {
public static void main(String[] args) {
int topics = 1800;
int partitionsPerTopic = 6;
int replicationFactor = 3;
int totalPartitions = topics * partitionsPerTopic;
int totalReplicas = totalPartitions * replicationFactor;
System.out.println("Topics: " + topics);
System.out.println("Partitions per topic: " + partitionsPerTopic);
System.out.println("Replication factor: " + replicationFactor);
System.out.println("Total partitions: " + totalPartitions);
System.out.println("Total replicas: " + totalReplicas);
System.out.println();
System.out.println("Each partition consumes:");
System.out.println(" - 1 directory on disk");
System.out.println(" - 2+ file handles (log + index)");
System.out.println(" - Controller metadata");
System.out.println(" - Memory for page cache");
System.out.println();
System.out.println("Plus for each topic:");
System.out.println(" - Creation approval from platform team");
System.out.println(" - ACL entries for producers and consumers");
System.out.println(" - Monitoring dashboards and alerts");
System.out.println(" - Retention policy configuration");
}
}
Expected output:
Topics: 1800
Partitions per topic: 6
Replication factor: 3
Total partitions: 10800
Total replicas: 32400
Each partition consumes:
- 1 directory on disk
- 2+ file handles (log + index)
- Controller metadata
- Memory for page cache
Plus for each topic:
- Creation approval from platform team
- ACL entries for producers and consumers
- Monitoring dashboards and alerts
- Retention policy configuration
Why is change so painful with proliferated topics?
Routing is baked into topic structure, which is the hardest thing to change in a running Kafka estate. If the business adds a fourth size band (extra-large) or a new country (SG), you must create a whole new set of topics, update the producer routing logic, add ACLs, and tell all consumers about the new topics.
At National Bank of India, when the compliance team requested a new "jumbo" transaction size band, the Kafka platform team had to create 200 new topics, update the producer's routing table, add ACLs for three consumer groups, and update the monitoring dashboards. With Solace, it would have been a single new subscription pattern.
package com.bank.kafka;
public class ChangeIsPainful {
public static void main(String[] args) {
System.out.println("=== Adding 'jumbo' size band to Kafka ===");
System.out.println("Step 1: Create 200 new topics (3 types x 50 cities x 1 new size)");
System.out.println("Step 2: Update producer routing table to include 'jumbo'");
System.out.println("Step 3: Add ACLs for each new topic");
System.out.println("Step 4: Update all consumer subscriptions");
System.out.println("Step 5: Add monitoring for 200 new topics");
System.out.println("Effort: days of platform team work");
System.out.println();
System.out.println("=== Adding 'jumbo' size band to Solace ===");
System.out.println("Step 1: Publisher starts using txn/IN/Mumbai/savings/jumbo");
System.out.println("Step 2: Consumer subscribes: txn/IN/*/savings/jumbo");
System.out.println("Effort: minutes, no broker changes needed");
}
}
Expected output:
=== Adding 'jumbo' size band to Kafka ===
Step 1: Create 200 new topics (3 types x 50 cities x 1 new size)
Step 2: Update producer routing table to include 'jumbo'
Step 3: Add ACLs for each new topic
Step 4: Update all consumer subscriptions
Step 5: Add monitoring for 200 new topics
Effort: days of platform team work
=== Adding 'jumbo' size band to Solace ===
Step 1: Publisher starts using txn/IN/Mumbai/savings/jumbo
Step 2: Consumer subscribes: txn/IN/*/savings/jumbo
Effort: minutes, no broker changes needed
How do teams mitigate topic proliferation in Kafka?
Teams use three common mitigations, none of which is free.
Mitigation 1: Coarse topics with consumer-side filtering. Keep one bank.transactions topic and let each consumer filter. Trades topic sprawl for wasted bandwidth and CPU.
Mitigation 2: Stream processing branching. Use Kafka Streams or ksqlDB to read the coarse topic and dynamically branch into narrower topics. Centralizes routing but adds a stateful service to operate.
Mitigation 3: Hybrid approach. A few high-value slices get their own topics; the long tail is filtered client-side.
package com.bank.kafka;
public class MitigationStrategies {
public static void main(String[] args) {
System.out.println("Mitigation 1: Coarse topic + consumer filtering");
System.out.println(" Pros: one topic, simple");
System.out.println(" Cons: every consumer reads everything, wastes bandwidth");
System.out.println();
System.out.println("Mitigation 2: Kafka Streams branching");
System.out.println(" Pros: precise routing, narrow output topics");
System.out.println(" Cons: extra stateful service to deploy and monitor");
System.out.println();
System.out.println("Mitigation 3: Hybrid");
System.out.println(" Pros: balances precision and simplicity");
System.out.println(" Cons: some consumers still filter, some topics still created");
System.out.println();
System.out.println("None of these is as clean as Solace wildcard subscriptions.");
}
}
Expected output:
Mitigation 1: Coarse topic + consumer filtering
Pros: one topic, simple
Cons: every consumer reads everything, wastes bandwidth
Mitigation 2: Kafka Streams branching
Pros: precise routing, narrow output topics
Cons: extra stateful service to deploy and monitor
Mitigation 3: Hybrid
Pros: balances precision and simplicity
Cons: some consumers still filter, some topics still created
None of these is as clean as Solace wildcard subscriptions.
How does Solace avoid topic proliferation?
Solace avoids the problem by making the topic a publish-time string in a single hierarchical namespace rather than a pre-created broker object. There is no createTopic step for each combination. The publisher writes a richly structured topic string, and routing is entirely a subscriber concern expressed with wildcards.
Adding a new routing dimension value (new city, new type) requires no broker change. Any topic string is instantly routable because the broker's trie-based matcher handles arbitrary combinations without pre-provisioning.
package com.bank.solace;
public class SolaceAvoidance {
public static void main(String[] args) {
System.out.println("=== Solace: No topic proliferation ===");
System.out.println();
System.out.println("Publisher uses any topic string:");
System.out.println(" txn/IN/Mumbai/savings/large");
System.out.println(" txn/IN/Delhi/current/small");
System.out.println(" txn/SG/Singapore/savings/jumbo <- new country + size!");
System.out.println();
System.out.println("No topics created. No broker changes.");
System.out.println();
System.out.println("Subscribers just adjust patterns:");
System.out.println(" txn/IN/> (all India)");
System.out.println(" txn/SG/> (all Singapore -- new!)");
System.out.println(" txn/*/*/savings/jumbo (all jumbo savings -- new!)");
System.out.println();
System.out.println("Adding a new dimension value: 0 topics created");
System.out.println("Adding a new subscriber: 0 topics created");
System.out.println("Total broker objects: unchanged");
}
}
Expected output:
=== Solace: No topic proliferation ===
Publisher uses any topic string:
txn/IN/Mumbai/savings/large
txn/IN/Delhi/current/small
txn/SG/Singapore/savings/jumbo <- new country + size!
No topics created. No broker changes.
Subscribers just adjust patterns:
txn/IN/> (all India)
txn/SG/> (all Singapore -- new!)
txn/*/*/savings/jumbo (all jumbo savings -- new!)
Adding a new dimension value: 0 topics created
Adding a new subscriber: 0 topics created
Total broker objects: unchanged
How do Kafka and Solace compare on topic management?
| Aspect | Kafka | Solace |
|---|---|---|
| Topic creation | Explicit, pre-created per combination | None, topic is a publish-time string |
| Topic count growth | Multiplicative (dimensions product) | Zero growth (one namespace) |
| Adding new dimension value | Create new topics + rewire producers + ACLs | Publisher uses new string, no changes |
| Adding new consumer | Subscribe to existing topic(s) | Register a new wildcard pattern |
| Partition cost | Each topic has partitions (memory, handles) | No partition concept for routing |
| Operational overhead | High: creation, ACLs, monitoring per topic | Low: one namespace, wildcard patterns |
| Change agility | Days (platform team approval) | Minutes (consumer-side change) |
package com.bank.comparison;
public class TopicManagementComparison {
public static void main(String[] args) {
System.out.println("Topic management comparison:");
System.out.println();
System.out.println("Kafka: 1800 topics x 6 partitions = 10800 partitions");
System.out.println(" Creating: platform team approval for each");
System.out.println(" Changing: days of work for one new dimension value");
System.out.println();
System.out.println("Solace: 0 topics to create");
System.out.println(" Creating: publisher sends any topic string");
System.out.println(" Changing: consumer adjusts subscription pattern");
}
}
Expected output:
Topic management comparison:
Kafka: 1800 topics x 6 partitions = 10800 partitions
Creating: platform team approval for each
Changing: days of work for one new dimension value
Solace: 0 topics to create
Creating: publisher sends any topic string
Changing: consumer adjusts subscription pattern
What are common mistakes?
Wrong: creating hundreds of Kafka topics to match Solace-like routing.
This leads to operational nightmares: partition explosion, ACL management for every topic, monitoring dashboards that cannot fit on a screen, and days of work every time a dimension changes.
package com.bank.kafka;
public class WrongTopicExplosion {
public static void main(String[] args) {
System.out.println("WRONG: Creating a topic per routing combination");
System.out.println(" txn.in.mumbai.savings.large (topic 1 of 1800)");
System.out.println(" txn.in.mumbai.savings.medium (topic 2 of 1800)");
System.out.println(" ... 1798 more topics");
System.out.println(" Result: 10800 partitions, ACL nightmare, brittle routing");
System.out.println();
System.out.println("CORRECT options:");
System.out.println(" A) Use Solace for fine-grained routing");
System.out.println(" B) Use one Kafka topic + consumer filtering");
System.out.println(" C) Use Kafka Streams to branch into a few key topics");
System.out.println(" D) Hybrid: Solace routes live, Kafka stores durably");
}
}
Expected output:
WRONG: Creating a topic per routing combination
txn.in.mumbai.savings.large (topic 1 of 1800)
txn.in.mumbai.savings.medium (topic 2 of 1800)
... 1798 more topics
Result: 10800 partitions, ACL nightmare, brittle routing
CORRECT options:
A) Use Solace for fine-grained routing
B) Use one Kafka topic + consumer filtering
C) Use Kafka Streams to branch into a few key topics
D) Hybrid: Solace routes live, Kafka stores durably
What should a senior engineer know?
The senior insight is that topic proliferation is a genuine architectural trade-off, not a bug.
- Kafka's flat topic model is what gives it unmatched sequential throughput, ordered partitions, long retention, and replay. The simplicity is the feature.
- Solace's hierarchical routing avoids proliferation but pays with broker-side matching CPU and a different retention and replay model.
- The cleanest answer to proliferation is often not to force Kafka to route finely, but to let Solace route and let Kafka store.
- A Kafka cluster's practical partition limit is around 200,000. With 6 partitions per topic and RF=3, that is around 11,000 topics before stress.
- KRaft mode (replacing ZooKeeper) raises the partition ceiling significantly, but proliferation is still an operational burden.
package com.bank.comparison;
public class SeniorProliferationInsights {
public static void main(String[] args) {
System.out.println("Senior insights on topic proliferation:");
System.out.println("1. Kafka's flat model = throughput + replay (feature, not bug)");
System.out.println("2. Solace's hierarchy = routing flexibility (different trade-off)");
System.out.println("3. Practical Kafka limit: ~200K partitions per cluster");
System.out.println("4. KRaft raises ceiling but proliferation is still an ops burden");
System.out.println("5. Best answer: Solace routes, Kafka stores");
System.out.println("6. Never force Kafka to be a content router");
}
}
Expected output:
Senior insights on topic proliferation:
1. Kafka's flat model = throughput + replay (feature, not bug)
2. Solace's hierarchy = routing flexibility (different trade-off)
3. Practical Kafka limit: ~200K partitions per cluster
4. KRaft raises ceiling but proliferation is still an ops burden
5. Best answer: Solace routes, Kafka stores
6. Never force Kafka to be a content router
Visual Overview
Topic Proliferation in Kafka: Routing Dimensions: Type: credit, debit, transfer (3) City: Mumbai, Delhi, ... (50 cities) (50) Size: small, medium, large (3) Topics = 3 x 50 x 3 = 450 topics! Partitions = 450 x 6 = 2,700 partitions Replicas = 2,700 x 3 = 8,100 replicas ┌──────────────────────────────────────┐ │ Kafka Cluster │ │ txn.credit.mumbai.small (6 parts)│ │ txn.credit.mumbai.medium (6 parts)│ │ txn.credit.mumbai.large (6 parts)│ │ txn.credit.delhi.small (6 parts)│ │ ... 446 more topics ... │ │ Total: 2,700 partitions │ └──────────────────────────────────────┘ Adding 1 new size band = 150 new topics! ───────────────────────────────────────── Solace: Zero Proliferation ┌──────────────────────────────────────┐ │ Solace Broker │ │ One hierarchical namespace. │ │ Zero pre-created topics. │ │ │ │ Publisher sends: │ │ txn/IN/Mumbai/savings/large │ │ txn/IN/Delhi/current/small │ │ txn/IN/Mumbai/savings/jumbo NEW! │ │ │ │ Subscriber patterns: │ │ txn/IN/> │ │ txn/IN/*/savings/large │ │ txn/IN/*/savings/jumbo NEW! │ └──────────────────────────────────────┘ Adding 1 new size band = 0 new topics!
Key Takeaway
Topic proliferation is the operational problem in Kafka where the number of topics grows multiplicatively with routing dimensions. Because the Kafka broker routes only at the granularity of a whole flat topic and never looks inside a record, encoding routing into topics means 3 types times 50 cities times 3 sizes produces 450 topics, each requiring creation, partition sizing, replication, ACLs, and monitoring. Worse, adding one new dimension value means creating a whole new batch of topics and rewiring producers.
Solace avoids this entirely because its topic is a publish-time string in a single hierarchical namespace, not a pre-created broker object. There is no per-combination creation step. The publisher writes a structured topic string like txn/IN/Mumbai/savings/large, and routing is purely a subscriber concern expressed with wildcards. A new city, a new size band, or a new subscriber is just a new string or pattern with zero broker changes.
At National Bank of India, forcing Kafka to route finely across all transaction dimensions would have meant creating and governing 1800 topics and rebuilding them every time compliance requested a new category. Instead, Pankaj let Solace handle the fine-grained live routing through one hierarchical namespace, while Kafka served as the durable backbone with a single bank.transactions topic retained for seven years.
Remember one sentence: topic proliferation happens because Kafka's routing granularity is the topic itself, and Solace avoids it because its routing granularity is the subscription pattern.
---