Back to Kafka Fundamentals: Topics, Partitions, Offsets
Q3Question 3 of 18

Give an example of Solace's hierarchical routing: publishing to `ps/news/{type}/{country}/{city}` and subscribing with patterns like `ps/news/*/BR/>` or `ps/news/SPORT/>`: and why is this hard in Kafka?

Key Concepts
  • Solace topics are hierarchical strings like txn/IN/Mumbai/savings/large, not pre-created broker objects.
  • The single-level wildcard (*) matches exactly one level; the multi-level wildcard (>) matches one or more remaining levels.
  • The Solace broker evaluates every message against active subscriptions and delivers only matches.
  • In Kafka, a topic is a flat pre-created log and the broker does no content-based routing.
  • Reproducing Solace routing in Kafka requires many fine-grained topics, consumer-side filtering, or a stream-processing layer.
  • Each Kafka workaround has a cost: topic explosion, wasted network bandwidth, or extra infrastructure.
  • The Solace broker uses a trie-based matcher for near-constant matching time per topic level.

Definition

In Solace the broker is smart and routes each message by its hierarchical topic pattern; in Kafka the broker is simple and the consumer must do the filtering itself.

Solace hierarchical routing lets a publisher encode routing dimensions directly into a slash-separated topic string, and subscribers use wildcards to express interest.

  • Hierarchical topic – a publish-time string like txn/IN/Mumbai/savings/large where each level carries one routing dimension.
  • **Single-level wildcard (*)** – matches exactly one level, so txn/*/Mumbai/> matches any country with Mumbai as the city.
  • Multi-level wildcard (>) – matches one or more remaining levels at the end, so txn/IN/> matches everything under India.
  • Broker-side matching – the Solace broker evaluates all subscriptions per message and delivers only to matching consumers.
  • Kafka flat topic – a pre-created partitioned log where the broker delivers all records to the consumer with no filtering.

Imagine National Bank of India has a single pneumatic tube network for transaction alerts. Every alert is labelled with a structured address: country, city, account type, and size. Each desk tells the mailroom a pattern: "send me anything from India, Mumbai, any type" or "send me all large savings from any city." The mailroom reads each label and routes the alert only to the matching desks. Nobody receives alerts they did not ask for, and adding a new desk is just registering a new pattern. In Kafka, every desk gets every alert and sorts through the pile themselves.

Understanding the Concepts

What is the example scenario?

Consider a transaction alerting system at National Bank of India. The system publishes transaction events where each event is tagged with a country, city, account type, and transaction size. Different desks want different slices of this feed.

The topic hierarchy is: txn/{country}/{city}/{accountType}/{size}

  • Example published topics:
  • txn/IN/Mumbai/savings/large (large savings transaction in Mumbai)
  • txn/IN/Delhi/current/small (small current-account transaction in Delhi)
  • txn/BR/SaoPaulo/savings/medium (medium savings transaction in Sao Paulo)
JAVA
package com.bank.solace;

import com.solacesystems.jcsmp.JCSMPFactory;
import com.solacesystems.jcsmp.JCSMPSession;
import com.solacesystems.jcsmp.JCSMPProperties;
import com.solacesystems.jcsmp.Topic;
import com.solacesystems.jcsmp.XMLMessageProducer;
import com.solacesystems.jcsmp.TextMessage;
import java.math.BigDecimal;

/**
 * Rajesh publishes transactions with hierarchical topics.
 * Each transaction carries its routing dimensions in the topic string.
 */
public class TransactionAlertPublisher {

    public static void main(String[] args) throws Exception {
        JCSMPProperties properties = new JCSMPProperties();
        properties.setProperty(JCSMPProperties.HOST, "solace-broker:55555");
        properties.setProperty(JCSMPProperties.VPN_NAME, "nbi-vpn");
        properties.setProperty(JCSMPProperties.USERNAME, "rajesh");

        JCSMPSession session = JCSMPFactory.onlyInstance().createSession(properties);
        session.connect();
        XMLMessageProducer producer = session.getMessageProducer(null);

        // Publish three transactions with different hierarchical topics
        String[][] events = {
            {"txn/IN/Mumbai/savings/large", "SAV2001", "750000.00"},
            {"txn/IN/Delhi/current/small", "CUR3001", "5000.00"},
            {"txn/BR/SaoPaulo/savings/medium", "SAV4001", "150000.00"}
        };

        for (String[] event : events) {
            Topic topic = JCSMPFactory.onlyInstance().createTopic(event[0]);
            TextMessage msg = JCSMPFactory.onlyInstance().createMessage(TextMessage.class);
            msg.setText("{\"account\":\"" + event[1]
                + "\",\"amount\":" + new BigDecimal(event[2]) + "}");
            producer.send(msg, topic);
            System.out.println("Published: " + event[0]);
        }

        session.closeSession();
    }
}
TEXT
Expected output:
Published: txn/IN/Mumbai/savings/large
Published: txn/IN/Delhi/current/small
Published: txn/BR/SaoPaulo/savings/medium

How does the single-level wildcard (*) work in this example?

The single-level wildcard * matches exactly one topic level. It is useful when you want to pin some dimensions and leave one open.

Pankaj wants all large savings transactions from any city in India. His subscription is txn/IN/*/savings/large. The * in the third position matches any city.

  • Matches txn/IN/Mumbai/savings/large (city = Mumbai)
  • Matches txn/IN/Delhi/savings/large (city = Delhi)
  • Does NOT match txn/IN/Mumbai/savings/small (size is small, not large)
  • Does NOT match txn/IN/Mumbai/current/large (type is current, not savings)
JAVA
package com.bank.solace;

import com.solacesystems.jcsmp.JCSMPFactory;
import com.solacesystems.jcsmp.JCSMPSession;
import com.solacesystems.jcsmp.JCSMPProperties;
import com.solacesystems.jcsmp.Topic;

/**
 * Pankaj subscribes with single-level wildcard: txn/IN/*/savings/large
 * Matches all large savings transactions in India regardless of city.
 */
public class SingleLevelWildcardDemo {

    public static void main(String[] args) throws Exception {
        JCSMPProperties properties = new JCSMPProperties();
        properties.setProperty(JCSMPProperties.HOST, "solace-broker:55555");
        properties.setProperty(JCSMPProperties.VPN_NAME, "nbi-vpn");
        properties.setProperty(JCSMPProperties.USERNAME, "pankaj");

        JCSMPSession session = JCSMPFactory.onlyInstance().createSession(properties);
        session.connect();

        // Subscribe: any city in India, savings, large
        Topic subscription = JCSMPFactory.onlyInstance().createTopic("txn/IN/*/savings/large");
        session.addSubscription(subscription);

        System.out.println("Subscription: txn/IN/*/savings/large");
        System.out.println();
        System.out.println("txn/IN/Mumbai/savings/large  -> MATCH (city=Mumbai)");
        System.out.println("txn/IN/Delhi/savings/large   -> MATCH (city=Delhi)");
        System.out.println("txn/IN/Mumbai/savings/small  -> NO MATCH (size=small)");
        System.out.println("txn/IN/Mumbai/current/large  -> NO MATCH (type=current)");
        System.out.println("txn/BR/SaoPaulo/savings/large -> NO MATCH (country=BR)");

        session.closeSession();
    }
}
TEXT
Expected output:
Subscription: txn/IN/*/savings/large

txn/IN/Mumbai/savings/large  -> MATCH (city=Mumbai)
txn/IN/Delhi/savings/large   -> MATCH (city=Delhi)
txn/IN/Mumbai/savings/small  -> NO MATCH (size=small)
txn/IN/Mumbai/current/large  -> NO MATCH (type=current)
txn/BR/SaoPaulo/savings/large -> NO MATCH (country=BR)

How does the multi-level wildcard (>) work in this example?

The multi-level wildcard > matches one or more remaining levels and can only appear at the end. It is useful when you want everything under a prefix regardless of depth.

Priya wants all transactions from India, any city, any type, any size. Her subscription is txn/IN/>. The > matches Mumbai/savings/large, Delhi/current/small, and any deeper path.

  • Matches txn/IN/Mumbai/savings/large (three levels after IN)
  • Matches txn/IN/Delhi/current/small (three levels after IN)
  • Does NOT match txn/BR/SaoPaulo/savings/medium (country is BR, not IN)
JAVA
package com.bank.solace;

import com.solacesystems.jcsmp.JCSMPFactory;
import com.solacesystems.jcsmp.JCSMPSession;
import com.solacesystems.jcsmp.JCSMPProperties;
import com.solacesystems.jcsmp.Topic;

/**
 * Priya subscribes with multi-level wildcard: txn/IN/>
 * Matches ALL transactions from India regardless of city, type, or size.
 */
public class MultiLevelWildcardDemo {

    public static void main(String[] args) throws Exception {
        JCSMPProperties properties = new JCSMPProperties();
        properties.setProperty(JCSMPProperties.HOST, "solace-broker:55555");
        properties.setProperty(JCSMPProperties.VPN_NAME, "nbi-vpn");
        properties.setProperty(JCSMPProperties.USERNAME, "priya");

        JCSMPSession session = JCSMPFactory.onlyInstance().createSession(properties);
        session.connect();

        // Subscribe: everything under India at any depth
        Topic subscription = JCSMPFactory.onlyInstance().createTopic("txn/IN/>");
        session.addSubscription(subscription);

        System.out.println("Subscription: txn/IN/>");
        System.out.println();
        System.out.println("txn/IN/Mumbai/savings/large     -> MATCH");
        System.out.println("txn/IN/Delhi/current/small      -> MATCH");
        System.out.println("txn/IN/Mumbai/savings/large/vip -> MATCH (deeper path)");
        System.out.println("txn/BR/SaoPaulo/savings/medium  -> NO MATCH (country=BR)");

        session.closeSession();
    }
}
TEXT
Expected output:
Subscription: txn/IN/>

txn/IN/Mumbai/savings/large     -> MATCH
txn/IN/Delhi/current/small      -> MATCH
txn/IN/Mumbai/savings/large/vip -> MATCH (deeper path)
txn/BR/SaoPaulo/savings/medium  -> NO MATCH (country=BR)

How can one message match multiple subscriptions?

A single message can match multiple subscriptions and be delivered to multiple consumers. The broker evaluates all active subscriptions independently.

If Priya subscribes to txn/IN/> and Pankaj subscribes to txn/IN/*/savings/large, then the message txn/IN/Mumbai/savings/large matches both and is delivered to both.

JAVA
package com.bank.solace;

public class MultipleMatchDemo {
    public static void main(String[] args) {
        String message = "txn/IN/Mumbai/savings/large";
        System.out.println("Published: " + message);
        System.out.println();
        System.out.println("Priya's subscription: txn/IN/>            -> MATCH");
        System.out.println("Pankaj's subscription: txn/IN/*/savings/large -> MATCH");
        System.out.println("Rajesh's subscription: txn/BR/>           -> NO MATCH");
        System.out.println();
        System.out.println("Delivered to: Priya and Pankaj (not Rajesh)");
        System.out.println("Broker evaluates all subscriptions per message.");
    }
}
TEXT
Expected output:
Published: txn/IN/Mumbai/savings/large

Priya's subscription: txn/IN/>            -> MATCH
Pankaj's subscription: txn/IN/*/savings/large -> MATCH
Rajesh's subscription: txn/BR/>           -> NO MATCH

Delivered to: Priya and Pankaj (not Rajesh)
Broker evaluates all subscriptions per message.

Why is this hard to do in Kafka?

Kafka's broker is deliberately simple. A topic is a flat, pre-created, partitioned log. The broker does not interpret the topic name structure and does not filter by message content. A consumer subscribes to a whole topic and reads every record.

To reproduce the Solace behavior in Kafka, you have three options, each with a cost.

JAVA
package com.bank.kafka;

public class WhyHardInKafka {
    public static void main(String[] args) {
        System.out.println("=== Option 1: Many fine-grained topics ===");
        System.out.println("Create: txn.in.mumbai.savings.large");
        System.out.println("Create: txn.in.mumbai.savings.small");
        System.out.println("Create: txn.in.delhi.current.large");
        System.out.println("... 3 types x 50 cities x 3 sizes = 450 topics");
        System.out.println("Cost: topic explosion, operational overhead");
        System.out.println();
        System.out.println("=== Option 2: One broad topic, consumer filters ===");
        System.out.println("Topic: bank.transactions (everything)");
        System.out.println("Consumer: if (city.equals('Mumbai') && size.equals('large'))");
        System.out.println("Cost: every consumer downloads and scans everything");
        System.out.println();
        System.out.println("=== Option 3: Kafka Streams branching ===");
        System.out.println("Read: bank.transactions");
        System.out.println("Branch by city/type/size into narrower topics");
        System.out.println("Cost: extra stateful infrastructure to operate");
    }
}
TEXT
Expected output:
=== Option 1: Many fine-grained topics ===
Create: txn.in.mumbai.savings.large
Create: txn.in.mumbai.savings.small
Create: txn.in.delhi.current.large
... 3 types x 50 cities x 3 sizes = 450 topics
Cost: topic explosion, operational overhead

=== Option 2: One broad topic, consumer filters ===
Topic: bank.transactions (everything)
Consumer: if (city.equals('Mumbai') && size.equals('large'))
Cost: every consumer downloads and scans everything

=== Option 3: Kafka Streams branching ===
Read: bank.transactions
Branch by city/type/size into narrower topics
Cost: extra stateful infrastructure to operate

How does the Solace broker match subscriptions efficiently?

The Solace broker maintains a trie (prefix tree) data structure keyed on topic levels. When a message arrives, the broker walks the trie from the root, matching each level of the published topic against the subscription tree. This gives near-constant matching time per level, regardless of how many subscriptions are active.

This is why adding a new subscription is cheap: it is just inserting a new path in the trie, not creating a new broker object like a Kafka topic.

JAVA
package com.bank.solace;

public class TrieMatchingDemo {
    public static void main(String[] args) {
        System.out.println("Solace Trie Matching:");
        System.out.println();
        System.out.println("Subscription trie:");
        System.out.println("  txn");
        System.out.println("  ├── IN");
        System.out.println("  │   ├── > (Priya: all India)");
        System.out.println("  │   └── * (any city)");
        System.out.println("  │       └── savings");
        System.out.println("  │           └── large (Pankaj: large savings India)");
        System.out.println("  └── BR");
        System.out.println("      └── > (Rajesh: all Brazil)");
        System.out.println();
        System.out.println("Message: txn/IN/Mumbai/savings/large");
        System.out.println("Walk: txn -> IN -> Priya matches (>)");
        System.out.println("Walk: txn -> IN -> * -> savings -> large -> Pankaj matches");
        System.out.println("Walk: txn -> BR -> not taken (country != BR)");
        System.out.println("Result: delivered to Priya and Pankaj");
    }
}
TEXT
Expected output:
Solace Trie Matching:

Subscription trie:
  txn
  +-- IN
  |   +-- > (Priya: all India)
  |   +-- * (any city)
  |       +-- savings
  |           +-- large (Pankaj: large savings India)
  +-- BR
      +-- > (Rajesh: all Brazil)

Message: txn/IN/Mumbai/savings/large
Walk: txn -> IN -> Priya matches (>)
Walk: txn -> IN -> * -> savings -> large -> Pankaj matches
Walk: txn -> BR -> not taken (country != BR)
Result: delivered to Priya and Pankaj

What are common mistakes?

Wrong: placing the multi-level wildcard in the middle of a subscription.

The > wildcard can only appear at the end of a Solace subscription. Placing it in the middle is invalid.

JAVA
package com.bank.solace;

public class WildcardMistakes {
    public static void main(String[] args) {
        System.out.println("WRONG: txn/>/savings/large");
        System.out.println("  The > wildcard can only appear at the END.");
        System.out.println("  This subscription is invalid in Solace.");
        System.out.println();
        System.out.println("CORRECT: txn/*/savings/large");
        System.out.println("  Use * for exactly one level in the middle.");
        System.out.println();
        System.out.println("CORRECT: txn/IN/Mumbai/>");
        System.out.println("  Use > at the end for one or more remaining levels.");
        System.out.println();
        System.out.println("WRONG (Kafka): Pattern.compile('txn.IN.Mumbai.*')");
        System.out.println("  Kafka regex matches TOPIC NAMES, not message content.");
        System.out.println("  This subscribes to all topics matching the regex,");
        System.out.println("  then delivers ALL records from each matched topic.");
    }
}
TEXT
Expected output:
WRONG: txn/>/savings/large
  The > wildcard can only appear at the END.
  This subscription is invalid in Solace.

CORRECT: txn/*/savings/large
  Use * for exactly one level in the middle.

CORRECT: txn/IN/Mumbai/>
  Use > at the end for one or more remaining levels.

WRONG (Kafka): Pattern.compile('txn.IN.Mumbai.*')
  Kafka regex matches TOPIC NAMES, not message content.
  This subscribes to all topics matching the regex,
  then delivers ALL records from each matched topic.

What should a senior engineer know?

The key senior insight is where the filtering CPU cost lives and what each design trades.

  • In Solace, the broker pays the CPU cost for subscription matching. The broker must be sized for the number of subscriptions and the message rate. Consumers receive only what they need.
  • In Kafka, the consumer pays the CPU cost for filtering. The broker is simple and fast, which gives Kafka its massive throughput. Consumers may waste bandwidth on irrelevant records.
  • Solace's trie-based matcher handles arbitrary subscription combinations without pre-provisioning. Adding a new routing need is a new subscription, not a new broker object.
  • Kafka's replay and long retention are not Solace's primary strength. Solace offers persistent queues, but its model is optimized for live routing, not long-horizon reprocessing.
  • The practical architecture often runs both: Solace for live selective routing and Kafka for durable retention, bridged together.
JAVA
package com.bank.comparison;

public class SeniorInsights {
    public static void main(String[] args) {
        System.out.println("Where does filtering CPU live?");
        System.out.println("  Solace: in the broker (size broker for subscriptions)");
        System.out.println("  Kafka:  in the consumer (wastes bandwidth)");
        System.out.println();
        System.out.println("What does each trade?");
        System.out.println("  Solace: trades throughput for routing flexibility");
        System.out.println("  Kafka:  trades routing for throughput + replay");
        System.out.println();
        System.out.println("Practical answer: use both");
        System.out.println("  Solace routes live events to selective consumers");
        System.out.println("  Kafka stores every event for years with replay");
    }
}
TEXT
Expected output:
Where does filtering CPU live?
  Solace: in the broker (size broker for subscriptions)
  Kafka:  in the consumer (wastes bandwidth)

What does each trade?
  Solace: trades throughput for routing flexibility
  Kafka:  trades routing for throughput + replay

Practical answer: use both
  Solace routes live events to selective consumers
  Kafka stores every event for years with replay

Visual Overview

DIAGRAM
  Solace Hierarchical Routing Example:

  Published Topics:
  txn/IN/Mumbai/savings/large
  txn/IN/Delhi/current/small
  txn/BR/SaoPaulo/savings/medium
        │
        ▼
  ┌──────────────────────────┐
  │     Solace Broker         │
  │     (trie matcher)        │
  │                           │
  │  Active Subscriptions:    │
  │  Priya:  txn/IN/>         │──► Priya receives:
  │  Pankaj: txn/IN/*/sav/lg  │    txn/IN/Mumbai/savings/large
  │  Rajesh: txn/BR/>         │    txn/IN/Delhi/current/small
  └──────────────────────────┘
        │            │
        │            └──► Pankaj receives:
        │                 txn/IN/Mumbai/savings/large
        │
        └──► Rajesh receives:
             txn/BR/SaoPaulo/savings/medium

  ─────────────────────────────────────────────

  Same scenario in Kafka:

  Option 1: 450 topics          Option 2: 1 topic
  txn.in.mumbai.savings.large   bank.transactions
  txn.in.mumbai.savings.small     │
  txn.in.delhi.current.small      ├─ Consumer A: reads ALL, filters
  ... (450 total)                 ├─ Consumer B: reads ALL, filters
  Operational nightmare.          └─ Consumer C: reads ALL, filters
                                  Wasted bandwidth.

Key Takeaway

Solace hierarchical routing lets a publisher encode routing dimensions directly into a slash-separated topic string, and subscribers express interest with level-aware wildcards. The single-level wildcard * matches exactly one level, so txn/IN/*/savings/large matches any city in India with large savings. The multi-level wildcard > matches one or more remaining levels, so txn/IN/> matches everything from India at any depth. The broker evaluates every message against all active subscriptions using a trie-based matcher and delivers only matching messages.

This is hard to reproduce in Kafka because Kafka's broker is deliberately simple. A topic is a flat, pre-created, partitioned log, the broker does no content-based routing, and a consumer reads every record in its assigned partitions. To get Solace-like behavior in Kafka you need many fine-grained topics (which explodes topic count), consumer-side filtering (which wastes bandwidth), or a stream-processing layer (which adds infrastructure).

At National Bank of India, Priya subscribed to txn/IN/> and received all India transactions. Pankaj subscribed to txn/IN/*/savings/large and received only large savings across all Indian cities. A single Mumbai large savings event matched both subscriptions and was delivered to both. Adding a new desk was just registering a new subscription pattern, with no topics to create and no producer to modify.

Remember one sentence: Solace's broker matches each message's hierarchical topic against wildcard subscriptions and delivers only to interested consumers, while Kafka delivers everything and expects consumers to filter on their own.

---

Share
Use ← → arrow keys to navigate