Definition
Yes, Kafka and Solace are frequently used together because they solve different problems and complement each other.
- Kafka as system of record – a partitioned, append-only log that retains events for the long term and supports deterministic replay, event sourcing, and stream processing.
- Solace as distribution layer – a broker-native hierarchical router that delivers precisely filtered slices to many heterogeneous consumers with low latency and no topic creation.
- Bridge – a connector or adapter that moves events between the two systems, ensuring every routed event also has a durable home and every stored event can be distributed.
- System of record – one authoritative source of truth (usually Kafka) so if the two diverge you know which to trust.
- Complementary design – each tool handles what its model does best, avoiding the pitfalls of forcing one into the other's role.
At National Bank of India, Solace is the smart mail room that instantly sorts each incoming transaction alert to exactly the desks that asked for that kind of alert, as desks keep changing their requests. Kafka is the permanent records vault where a copy of every alert is filed in order and kept for years, so auditors can replay the entire history whenever they need. The mail room makes sure the right person gets the right alert now; the vault makes sure nothing is ever lost and everything can be reconstructed later. A courier (the bridge) makes sure every alert that passes through the mail room also gets filed in the vault.
Understanding the Concepts
Why are Kafka and Solace complementary?
Their design centers are different. Kafka optimizes for a durable, ordered, partitioned log with long retention, offset-based replay, per-key ordering, and a stream-processing ecosystem. Solace optimizes for broker-side hierarchical routing with wildcard subscriptions, low-latency fan-out, multi-protocol access, and dynamic filtering without topic creation.
When a system needs both durable history and flexible live routing, using each for its strength is cleaner than forcing one to do both.
package com.bank.architecture;
public class WhyComplementary {
public static void main(String[] args) {
System.out.println("=== Kafka's strengths ===");
System.out.println(" Durable retention (days/weeks/years)");
System.out.println(" Deterministic replay (reset offset)");
System.out.println(" Event sourcing (log = truth)");
System.out.println(" Stream processing (Kafka Streams, ksqlDB)");
System.out.println(" High sequential throughput");
System.out.println();
System.out.println("=== Solace's strengths ===");
System.out.println(" Hierarchical wildcard routing");
System.out.println(" Low-latency selective fan-out");
System.out.println(" Dynamic routing (no new topics needed)");
System.out.println(" Multi-protocol (JMS, MQTT, AMQP, REST)");
System.out.println();
System.out.println("Together: Solace routes live, Kafka stores durably.");
}
}
Expected output:
=== Kafka's strengths ===
Durable retention (days/weeks/years)
Deterministic replay (reset offset)
Event sourcing (log = truth)
Stream processing (Kafka Streams, ksqlDB)
High sequential throughput
=== Solace's strengths ===
Hierarchical wildcard routing
Low-latency selective fan-out
Dynamic routing (no new topics needed)
Multi-protocol (JMS, MQTT, AMQP, REST)
Together: Solace routes live, Kafka stores durably.
What are the integration patterns?
There are three common patterns for connecting Kafka and Solace.
Pattern 1: Solace-in-front, Kafka-behind. The producer publishes to Solace, where the broker routes slices to live consumers. A bridge persists the full stream into Kafka. Live consumers get only their slice; Kafka holds the complete record.
Pattern 2: Kafka-as-source, Solace-as-distribution. Events are produced into Kafka as the system of record, and a bridge republishes selected streams into Solace for dynamic distribution.
Pattern 3: Bidirectional bridge. Events flow both ways through a bridge, with each system serving its role.
package com.bank.architecture;
public class IntegrationPatterns {
public static void main(String[] args) {
System.out.println("Pattern 1: Solace-in-front, Kafka-behind");
System.out.println(" Producer -> Solace -> live desks (selective)");
System.out.println(" -> bridge -> Kafka (complete record)");
System.out.println();
System.out.println("Pattern 2: Kafka-as-source, Solace-as-distribution");
System.out.println(" Producer -> Kafka (system of record)");
System.out.println(" -> bridge -> Solace -> live desks (selective)");
System.out.println();
System.out.println("Pattern 3: Bidirectional bridge");
System.out.println(" Solace <-> bridge <-> Kafka");
System.out.println(" Each system serves its role");
}
}
Expected output:
Pattern 1: Solace-in-front, Kafka-behind
Producer -> Solace -> live desks (selective)
-> bridge -> Kafka (complete record)
Pattern 2: Kafka-as-source, Solace-as-distribution
Producer -> Kafka (system of record)
-> bridge -> Solace -> live desks (selective)
Pattern 3: Bidirectional bridge
Solace <-> bridge <-> Kafka
Each system serves its role
How do you build the bridge?
The bridge can be implemented with Kafka Connect using a Solace connector, with Solace's own bridging capabilities, or with a custom adapter.
At National Bank of India, Pankaj uses a Kafka Connect sink connector that consumes from Solace and produces to Kafka, ensuring every transaction event has a durable copy.
package com.bank.bridge;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import com.solacesystems.jcsmp.JCSMPFactory;
import com.solacesystems.jcsmp.JCSMPSession;
import com.solacesystems.jcsmp.JCSMPProperties;
import com.solacesystems.jcsmp.Topic;
import com.solacesystems.jcsmp.XMLMessageListener;
import com.solacesystems.jcsmp.BytesXMLMessage;
import com.solacesystems.jcsmp.TextMessage;
import com.solacesystems.jcsmp.XMLMessageConsumer;
import java.util.Properties;
/**
* Bridge: reads from Solace, writes to Kafka.
* Ensures every routed event also lands in the durable Kafka log.
*/
public class SolaceToKafkaBridge {
public static void main(String[] args) throws Exception {
// Kafka producer
Properties kafkaProps = new Properties();
kafkaProps.put("bootstrap.servers", "kafka-1:9092");
kafkaProps.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
kafkaProps.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
kafkaProps.put("acks", "all");
KafkaProducer<String, String> kafkaProducer = new KafkaProducer<>(kafkaProps);
// Solace consumer
JCSMPProperties solaceProps = new JCSMPProperties();
solaceProps.setProperty(JCSMPProperties.HOST, "solace-broker:55555");
solaceProps.setProperty(JCSMPProperties.VPN_NAME, "nbi-vpn");
solaceProps.setProperty(JCSMPProperties.USERNAME, "bridge-user");
JCSMPSession session = JCSMPFactory.onlyInstance().createSession(solaceProps);
session.connect();
// Subscribe to all transactions
Topic subscription = JCSMPFactory.onlyInstance().createTopic("txn/>");
session.addSubscription(subscription);
XMLMessageConsumer consumer = session.getMessageConsumer(new XMLMessageListener() {
@Override
public void onReceive(BytesXMLMessage message) {
if (message instanceof TextMessage) {
String text = ((TextMessage) message).getText();
String topic = message.getDestination().getName();
// Use Solace topic as Kafka key for partitioning
kafkaProducer.send(new ProducerRecord<>(
"bank.transactions", topic, text));
}
}
@Override
public void onException(com.solacesystems.jcsmp.JCSMPException e) {
System.err.println("Bridge error: " + e.getMessage());
}
});
consumer.start();
System.out.println("Bridge started: Solace txn/> -> Kafka bank.transactions");
System.out.println("Every routed event also lands in the durable Kafka log.");
}
}
Expected output:
Bridge started: Solace txn/> -> Kafka bank.transactions
Every routed event also lands in the durable Kafka log.
Why must you pick one system of record?
When two systems hold copies of the same data, they can diverge. A network glitch might cause the bridge to miss events, or Solace might age out messages that Kafka still retains. You need one authoritative source of truth.
At National Bank of India, Kafka is the system of record. If the Solace delivery and the Kafka record ever disagree, Kafka is authoritative and the team replays from Kafka to rebuild.
package com.bank.architecture;
public class SystemOfRecord {
public static void main(String[] args) {
System.out.println("=== System of Record: Kafka ===");
System.out.println();
System.out.println("Rule: Kafka is the single source of truth.");
System.out.println();
System.out.println("Scenario: bridge stalls for 5 minutes");
System.out.println(" Solace: delivered events to live desks (partial)");
System.out.println(" Kafka: missed 5 minutes of events");
System.out.println();
System.out.println("Recovery:");
System.out.println(" 1. Fix the bridge");
System.out.println(" 2. Bridge replays from Solace's persistent queue");
System.out.println(" 3. Kafka catches up (system of record is complete)");
System.out.println();
System.out.println("If Kafka and Solace disagree, trust Kafka.");
System.out.println("Replay from Kafka to rebuild any downstream view.");
}
}
Expected output:
=== System of Record: Kafka ===
Rule: Kafka is the single source of truth.
Scenario: bridge stalls for 5 minutes
Solace: delivered events to live desks (partial)
Kafka: missed 5 minutes of events
Recovery:
1. Fix the bridge
2. Bridge replays from Solace's persistent queue
3. Kafka catches up (system of record is complete)
If Kafka and Solace disagree, trust Kafka.
Replay from Kafka to rebuild any downstream view.
What about ordering and delivery guarantees across the bridge?
Kafka gives per-partition ordering keyed by account ID. When bridging from Solace to Kafka, you need to decide how the Solace topic hierarchy maps to Kafka keys and partitions.
The bridge should extract the account ID from the message or the Solace topic and use it as the Kafka key so all events for one account land in the same partition and stay ordered. The bridge must also handle duplicates idempotently, because network retries can produce duplicates.
package com.bank.bridge;
public class BridgeOrderingGuarantees {
public static void main(String[] args) {
System.out.println("Bridge ordering strategy:");
System.out.println();
System.out.println("Solace topic: txn/IN/Mumbai/savings/large");
System.out.println("Message payload: {account: SAV2001, amount: 750000.00}");
System.out.println();
System.out.println("Kafka key: SAV2001 (extracted from payload)");
System.out.println("Kafka topic: bank.transactions");
System.out.println("Kafka partition: hash(SAV2001) % numPartitions");
System.out.println();
System.out.println("Result: all events for SAV2001 are in one partition, ordered.");
System.out.println();
System.out.println("Idempotency: enable.idempotence=true on Kafka producer");
System.out.println("Prevents duplicate records from network retries.");
}
}
Expected output:
Bridge ordering strategy:
Solace topic: txn/IN/Mumbai/savings/large
Message payload: {account: SAV2001, amount: 750000.00}
Kafka key: SAV2001 (extracted from payload)
Kafka topic: bank.transactions
Kafka partition: hash(SAV2001) % numPartitions
Result: all events for SAV2001 are in one partition, ordered.
Idempotency: enable.idempotence=true on Kafka producer
Prevents duplicate records from network retries.
What about retention mismatches?
Solace may age out or drop messages faster than Kafka retains them. This is precisely why the durable copy belongs in Kafka. Kafka should retain events for the longest required period (often years for compliance), while Solace focuses on live delivery.
Events that match no live Solace subscription are still persisted to Kafka through the bridge, keeping the durable record complete regardless of who was listening at the time.
package com.bank.architecture;
public class RetentionMismatch {
public static void main(String[] args) {
System.out.println("Retention comparison:");
System.out.println();
System.out.println("Solace:");
System.out.println(" Direct messages: no persistence (fire and forget)");
System.out.println(" Persistent queues: configurable, typically hours/days");
System.out.println(" Purpose: live delivery, not long-term storage");
System.out.println();
System.out.println("Kafka:");
System.out.println(" retention.ms=220752000000 (7 years)");
System.out.println(" Purpose: system of record, compliance, replay");
System.out.println();
System.out.println("Rule: durable copy ALWAYS belongs in Kafka.");
System.out.println("Even events with no live Solace subscriber get stored.");
}
}
Expected output:
Retention comparison:
Solace:
Direct messages: no persistence (fire and forget)
Persistent queues: configurable, typically hours/days
Purpose: live delivery, not long-term storage
Kafka:
retention.ms=220752000000 (7 years)
Purpose: system of record, compliance, replay
Rule: durable copy ALWAYS belongs in Kafka.
Even events with no live Solace subscriber get stored.
What are common mistakes?
Wrong: running the bridge without monitoring it.
A stalled bridge silently breaks either durability (events not reaching Kafka) or live delivery (events not reaching Solace). The bridge must be monitored as a first-class component.
package com.bank.bridge;
public class BridgeMonitoringMistake {
public static void main(String[] args) {
System.out.println("WRONG: Deploy bridge and forget about it");
System.out.println(" Bridge stalls at 3 AM");
System.out.println(" Solace desks get events but Kafka does not");
System.out.println(" Nobody notices until compliance audit finds gaps");
System.out.println();
System.out.println("CORRECT: Monitor bridge as first-class infra");
System.out.println(" Alert on: bridge process health");
System.out.println(" Alert on: Kafka producer error rate");
System.out.println(" Alert on: Solace consumer lag");
System.out.println(" Alert on: bridge throughput dropping to zero");
}
}
Expected output:
WRONG: Deploy bridge and forget about it
Bridge stalls at 3 AM
Solace desks get events but Kafka does not
Nobody notices until compliance audit finds gaps
CORRECT: Monitor bridge as first-class infra
Alert on: bridge process health
Alert on: Kafka producer error rate
Alert on: Solace consumer lag
Alert on: bridge throughput dropping to zero
Wrong: treating Solace as the system of record.
Solace's retention and replay model is not designed for long-horizon compliance storage. Kafka's partitioned log with configurable retention is the right choice for the authoritative copy.
package com.bank.architecture;
public class WrongSystemOfRecord {
public static void main(String[] args) {
System.out.println("WRONG: Solace as system of record");
System.out.println(" Solace ages out messages in days");
System.out.println(" Cannot replay 6 months of transactions");
System.out.println(" Compliance cannot reconstruct the ledger");
System.out.println();
System.out.println("CORRECT: Kafka as system of record");
System.out.println(" Kafka retains events for 7 years");
System.out.println(" Reset offset to earliest and replay all");
System.out.println(" Compliance can reconstruct any period");
}
}
Expected output:
WRONG: Solace as system of record
Solace ages out messages in days
Cannot replay 6 months of transactions
Compliance cannot reconstruct the ledger
CORRECT: Kafka as system of record
Kafka retains events for 7 years
Reset offset to earliest and replay all
Compliance can reconstruct any period
What should a senior engineer know?
A senior engineer ensures the complementary architecture is operationally sound.
- Pick one system of record (Kafka) so you always know which copy is authoritative.
- The bridge must handle ordering: extract the account ID and use it as the Kafka key so per-account order is preserved.
- Enable idempotence (
enable.idempotence=true) on the Kafka producer in the bridge to prevent duplicate records from network retries. - Monitor the bridge with the same rigor as the brokers: throughput, error rate, lag, and process health.
- Retention mismatches are expected. Kafka retains for years; Solace focuses on live delivery. The durable copy belongs in Kafka.
- Events with no live Solace subscriber must still reach Kafka. The bridge subscribes to
txn/>(everything) so the durable record is complete.
package com.bank.architecture;
public class SeniorBridgeChecklist {
public static void main(String[] args) {
System.out.println("Senior bridge checklist:");
System.out.println("1. Kafka = system of record (authoritative)");
System.out.println("2. Bridge key = account ID (per-account order in Kafka)");
System.out.println("3. enable.idempotence=true (no duplicates)");
System.out.println("4. Monitor bridge: throughput, errors, lag, health");
System.out.println("5. Kafka retention >> Solace retention");
System.out.println("6. Bridge subscribes to txn/> (everything)");
System.out.println("7. Even unmatched events reach Kafka");
}
}
Expected output:
Senior bridge checklist:
1. Kafka = system of record (authoritative)
2. Bridge key = account ID (per-account order in Kafka)
3. enable.idempotence=true (no duplicates)
4. Monitor bridge: throughput, errors, lag, health
5. Kafka retention >> Solace retention
6. Bridge subscribes to txn/> (everything)
7. Even unmatched events reach Kafka
Visual Overview
Complementary Architecture at National Bank of India:
Transaction
Event ──────►┌──────────────────┐
│ Solace Broker │
│ (live routing) │
│ │
│ txn/IN/Mumbai/> │──► Mumbai Desk (selective)
│ txn/BR/> │──► Brazil Ops (selective)
│ txn/*/*/*/large │──► Large Txn Monitor
│ │
│ txn/> │──► Bridge (EVERYTHING)
└───────┬──────────┘
│
▼
┌──────────────────┐
│ Bridge │
│ Solace -> Kafka │
│ key = accountId │
│ idempotent=true │
└───────┬──────────┘
│
▼
┌──────────────────┐
│ Kafka Cluster │
│ (system of │
│ record) │
│ │
│ bank.transactions│
│ retention: 7 yrs │
│ │
│ ├─ Analytics │
│ ├─ Fraud (Streams)│
│ └─ Compliance │
│ (replay) │
└──────────────────┘
Key Takeaway
Yes, Kafka and Solace are very commonly used together, and the reason is that they solve different problems and complement each other rather than compete. Kafka is the durable system of record: a partitioned log that retains events for years and supports offset-based replay, event sourcing, and stream processing. Solace is the dynamic distribution layer: a hierarchical router that delivers precisely filtered slices to many consumers with low latency and without creating new topics.
The way they fit together is a bridge. A transaction event is published once, Solace routes it live to exactly the desks whose wildcard subscriptions match, and the bridge captures the same event into a Kafka topic retained for years. Live consumers get only their slice without topic proliferation, while Kafka keeps the complete, ordered, replayable record for compliance and analytics. Even events that match no live subscription are still persisted to Kafka because the bridge subscribes to everything.
At National Bank of India, Pankaj ran exactly this architecture. Trading desks wanted diverse, frequently changing slices of the live transaction feed, so Solace handled the selective routing. Compliance needed every transaction stored durably for seven years and replayable to rebuild the ledger, so a bridge captured every event into Kafka as the authoritative system of record. The bridge was monitored as first-class infrastructure with alerts on throughput, errors, and lag.
Remember one sentence: Solace makes sure the right desk gets the right event now, Kafka makes sure nothing is ever lost and the whole history can be replayed later, and a monitored bridge connects them.
---