Back to Kafka Streams
Q4Question 4 of 27

What is the difference between Kafka Streams and Kafka Connect

Answer: Kafka Streams processes data inside Kafka (read, transform, write to another topic), while Kafka Connect moves data between Kafka and external systems (database to Kafka, Kafka to Elasticsearch) using pre-built connectors.

Key Concepts
  • Kafka Streams processes data inside Kafka: it reads from a topic, transforms or aggregates, and writes to another topic.
  • Kafka Connect moves data between Kafka and external systems like databases, Elasticsearch, S3, or files.
  • Kafka Connect uses ready-made connectors (source and sink) so you do not write Java code for integration.
  • Kafka Streams is a Java library you embed in your application; Kafka Connect is a runtime framework with workers.
  • Use Kafka Streams when you need business logic: filtering, joining, aggregating, or windowing on Kafka data.
  • Use Kafka Connect when you need to ingest data into Kafka from a database or push Kafka data to an external store.
  • In a real pipeline, Kafka Connect ingests data into Kafka, Kafka Streams processes it, and Kafka Connect exports the result.

Definition

Kafka Streams processes data inside Kafka (read, transform, write to another topic), while Kafka Connect moves data between Kafka and external systems (database to Kafka, Kafka to Elasticsearch) using pre-built connectors.
  • Kafka Streams is a Java client library for stream processing; it reads from Kafka topics, applies business logic (filter, map, join, aggregate, window), and writes results back to Kafka topics.
  • Kafka Connect is a framework for moving data between Kafka and external systems without writing custom producer/consumer code; it uses plug-in connectors.
  • Source Connector reads data from an external system (MySQL, PostgreSQL, files, REST APIs) and writes it to a Kafka topic.
  • Sink Connector reads data from a Kafka topic and writes it to an external system (Elasticsearch, S3, MongoDB, HDFS).
  • Connector workers are the runtime processes that execute connectors; they can run in standalone mode (single process) or distributed mode (cluster of workers with automatic task balancing).

Think of National Bank of India's data pipeline. Kafka Connect is like the postal service that picks up cheques from branch offices (source) and delivers printed statements to customer mailboxes (sink). Kafka Streams is the processing department inside the bank that reads those cheques, validates amounts, calculates running balances, and flags suspicious transactions. The postal service moves documents in and out; the processing department applies business rules. You need both: Connect to move data, Streams to process it.

Understanding the Concepts

What is Kafka Streams?

Kafka Streams is a Java library for processing data that already lives in Kafka topics. You define a processing topology (a pipeline of operations), and Kafka Streams continuously reads records, applies your logic, and writes results to output topics.

Common operations include filtering bad records, transforming values, joining two streams, grouping by key and aggregating, and windowed computations.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;

import java.math.BigDecimal;

public class TransactionFilterTopology {
    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> transactions = builder.stream("bank-transactions");

        // Filter: keep only transactions above 50,000 rupees
        transactions
            .filter((accountId, value) -> {
                BigDecimal amount = new BigDecimal(value.split("\\|")[2]);
                return amount.compareTo(new BigDecimal("50000")) > 0;
            })
            .to("large-transactions");

        Topology topology = builder.build();
        System.out.println("Kafka Streams topology built");
        System.out.println("  Input:  bank-transactions");
        System.out.println("  Filter: amount > 50,000");
        System.out.println("  Output: large-transactions");
        return topology;
    }

    public static void main(String[] args) {
        build();
    }
}
TEXT
Expected output:
Kafka Streams topology built
  Input:  bank-transactions
  Filter: amount > 50,000
  Output: large-transactions

What is Kafka Connect?

Kafka Connect is a framework for moving data between Kafka and external systems. You do not write Java producer or consumer code. Instead, you configure a connector using JSON, and Connect workers handle the rest.

A source connector reads from an external system and writes to Kafka. A sink connector reads from Kafka and writes to an external system.

JAVA
package com.bank.connect;

public class KafkaConnectExample {
    public static void main(String[] args) {
        System.out.println("=== Source Connector Example ===");
        System.out.println("  Name:    postgres-source");
        System.out.println("  Class:   io.debezium.connector.postgresql.PostgresConnector");
        System.out.println("  DB:      National Bank of India - accounts table");
        System.out.println("  Topic:   bank.public.accounts");
        System.out.println("  Action:  Every INSERT/UPDATE/DELETE in accounts table");
        System.out.println("           becomes a Kafka message automatically");
        System.out.println();
        System.out.println("=== Sink Connector Example ===");
        System.out.println("  Name:    elasticsearch-sink");
        System.out.println("  Class:   io.confluent.connect.elasticsearch.ElasticsearchSinkConnector");
        System.out.println("  Topic:   large-transactions");
        System.out.println("  Target:  Elasticsearch index: large-transactions");
        System.out.println("  Action:  Every Kafka message is indexed in Elasticsearch");
        System.out.println("           for search and dashboards");
    }
}
TEXT
Expected output:
=== Source Connector Example ===
  Name:    postgres-source
  Class:   io.debezium.connector.postgresql.PostgresConnector
  DB:      National Bank of India - accounts table
  Topic:   bank.public.accounts
  Action:  Every INSERT/UPDATE/DELETE in accounts table
           becomes a Kafka message automatically

=== Sink Connector Example ===
  Name:    elasticsearch-sink
  Class:   io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
  Topic:   large-transactions
  Target:  Elasticsearch index: large-transactions
  Action:  Every Kafka message is indexed in Elasticsearch
           for search and dashboards

How do Kafka Streams and Kafka Connect differ?

FeatureKafka StreamsKafka Connect
PurposeProcess data inside KafkaMove data in/out of Kafka
InputKafka topicExternal system (source) or Kafka topic (sink)
OutputKafka topicKafka topic (source) or external system (sink)
Code requiredJava application with topologyJSON configuration (no Java code)
DeploymentRuns inside your application (library)Runs as a Connect worker cluster (framework)
Operationsfilter, map, join, aggregate, windowRead rows, write rows, schema conversion
StateBuilt-in state stores (RocksDB)No processing state; only offset tracking
ScalingBy Kafka partition countBy connector task count
Fault toleranceChangelog topics restore stateWorker rebalancing reassigns tasks
Exactly-onceprocessing.guarantee = exactly_once_v2Source: depends on connector; Sink: idempotent writes
Use caseBusiness logic on Kafka dataDatabase CDC, data lake ingestion, search indexing
JAVA
package com.bank.streams;

public class StreamsVsConnectComparison {
    public static void main(String[] args) {
        System.out.println("Kafka Streams:");
        System.out.println("  - Java library, runs in your app");
        System.out.println("  - Processes data: filter, join, aggregate");
        System.out.println("  - Kafka topic in, Kafka topic out");
        System.out.println("  - You write Java code");
        System.out.println();
        System.out.println("Kafka Connect:");
        System.out.println("  - Framework with workers, runs separately");
        System.out.println("  - Moves data: database to Kafka, Kafka to Elasticsearch");
        System.out.println("  - External system in/out of Kafka");
        System.out.println("  - You write JSON config, no Java code");
    }
}
TEXT
Expected output:
Kafka Streams:
  - Java library, runs in your app
  - Processes data: filter, join, aggregate
  - Kafka topic in, Kafka topic out
  - You write Java code

Kafka Connect:
  - Framework with workers, runs separately
  - Moves data: database to Kafka, Kafka to Elasticsearch
  - External system in/out of Kafka
  - You write JSON config, no Java code

How do Kafka Streams and Kafka Connect work together in a real banking pipeline?

In a production banking system, you typically use both. Kafka Connect ingests data into Kafka, Kafka Streams processes it, and Kafka Connect exports the result.

Project structure

DIAGRAM
bank-transaction-pipeline/
├── connect-configs/
│   ├── postgres-source.json        (source connector config)
│   └── elasticsearch-sink.json     (sink connector config)
├── streams-app/
│   ├── pom.xml
│   └── src/
│       └── main/
│           └── java/
│               └── com/
│                   └── bank/
│                       └── streams/
│                           ├── FraudDetectionApp.java       (main class)
│                           ├── FraudDetectionTopology.java  (topology)
│                           └── Transaction.java             (model)
└── docker-compose.yml

Transaction.java

JAVA
package com.bank.streams;

import java.math.BigDecimal;

public class Transaction {
    private String accountId;
    private String type;
    private BigDecimal amount;
    private String description;
    private String timestamp;

    public Transaction() {}

    public Transaction(String accountId, String type, BigDecimal amount,
                       String description, String timestamp) {
        this.accountId = accountId;
        this.type = type;
        this.amount = amount;
        this.description = description;
        this.timestamp = timestamp;
    }

    public String getAccountId() { return accountId; }
    public void setAccountId(String accountId) { this.accountId = accountId; }
    public String getType() { return type; }
    public void setType(String type) { this.type = type; }
    public BigDecimal getAmount() { return amount; }
    public void setAmount(BigDecimal amount) { this.amount = amount; }
    public String getDescription() { return description; }
    public void setDescription(String description) { this.description = description; }
    public String getTimestamp() { return timestamp; }
    public void setTimestamp(String timestamp) { this.timestamp = timestamp; }

    @Override
    public String toString() {
        return accountId + "|" + type + "|" + amount + "|" + description + "|" + timestamp;
    }

    public static Transaction fromString(String value) {
        String[] parts = value.split("\\|");
        return new Transaction(
            parts[0], parts[1], new BigDecimal(parts[2]), parts[3], parts[4]);
    }
}

postgres-source.json (Kafka Connect source config)

JSON
{
  "name": "postgres-accounts-source",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "bankadmin",
    "database.password": "secret",
    "database.dbname": "nationalbank",
    "database.server.name": "nbi",
    "table.include.list": "public.transactions",
    "topic.prefix": "nbi",
    "plugin.name": "pgoutput",
    "slot.name": "transaction_slot"
  }
}

FraudDetectionTopology.java (Kafka Streams processing)

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;

import java.math.BigDecimal;

public class FraudDetectionTopology {

    private static final BigDecimal FRAUD_THRESHOLD = new BigDecimal("500000");

    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        // Read transactions ingested by Kafka Connect
        KStream<String, String> transactions = builder.stream("nbi.public.transactions");

        // Fraud detection: flag transactions above 5,00,000 rupees
        KStream<String, String> suspicious = transactions
            .filter((key, value) -> {
                Transaction txn = Transaction.fromString(value);
                return txn.getAmount().compareTo(FRAUD_THRESHOLD) > 0;
            })
            .mapValues(value -> {
                Transaction txn = Transaction.fromString(value);
                txn.setDescription("SUSPICIOUS: " + txn.getDescription().toUpperCase());
                return txn.toString();
            });

        // Write flagged transactions for Kafka Connect to export
        suspicious.to("fraud-alerts");

        // Also write all transactions to a clean topic
        transactions.to("processed-transactions");

        Topology topology = builder.build();
        System.out.println("Fraud detection topology built");
        System.out.println("  Source:  nbi.public.transactions (from Kafka Connect)");
        System.out.println("  Filter:  amount > 500,000 INR");
        System.out.println("  Output:  fraud-alerts (Kafka Connect exports to Elasticsearch)");
        System.out.println("  Output:  processed-transactions");
        return topology;
    }

    public static void main(String[] args) {
        build();
    }
}
TEXT
Expected output:
Fraud detection topology built
  Source:  nbi.public.transactions (from Kafka Connect)
  Filter:  amount > 500,000 INR
  Output:  fraud-alerts (Kafka Connect exports to Elasticsearch)
  Output:  processed-transactions

elasticsearch-sink.json (Kafka Connect sink config)

JSON
{
  "name": "elasticsearch-fraud-sink",
  "config": {
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "topics": "fraud-alerts",
    "connection.url": "http://elasticsearch:9200",
    "type.name": "_doc",
    "key.ignore": "true",
    "schema.ignore": "true",
    "behavior.on.malformed.documents": "warn"
  }
}

FraudDetectionApp.java (main class)

JAVA
package com.bank.streams;

import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;

import java.util.Properties;
import java.util.concurrent.CountDownLatch;

public class FraudDetectionApp {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "fraud-detection-service");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, "exactly_once_v2");

        KafkaStreams streams = new KafkaStreams(
            FraudDetectionTopology.build(), props);
        CountDownLatch latch = new CountDownLatch(1);

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            streams.close();
            latch.countDown();
            System.out.println("Fraud detection service stopped");
        }));

        try {
            streams.start();
            System.out.println("Fraud detection service started");
            System.out.println("  Pipeline: PostgreSQL -> Connect -> Streams -> Connect -> Elasticsearch");
            latch.await();
        } catch (Exception e) {
            System.exit(1);
        }
    }
}
TEXT
Expected output:
Fraud detection topology built
  Source:  nbi.public.transactions (from Kafka Connect)
  Filter:  amount > 500,000 INR
  Output:  fraud-alerts (Kafka Connect exports to Elasticsearch)
  Output:  processed-transactions
Fraud detection service started
  Pipeline: PostgreSQL -> Connect -> Streams -> Connect -> Elasticsearch

How to run the complete pipeline

BASH
# Step 1: Start all services
docker-compose up -d zookeeper kafka postgres elasticsearch connect

# Step 2: Create the transactions table in PostgreSQL
psql -h localhost -U bankadmin -d nationalbank -c "
CREATE TABLE transactions (
    id SERIAL PRIMARY KEY,
    account_id VARCHAR(20) NOT NULL,
    type VARCHAR(10) NOT NULL,
    amount DECIMAL(15,2) NOT NULL,
    description VARCHAR(200) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);"

# Step 3: Register the source connector
curl -X POST http://localhost:8083/connectors \
  -H "Content-Type: application/json" \
  -d @connect-configs/postgres-source.json

# Step 4: Build and start the Kafka Streams app
cd streams-app
mvn clean package
java -cp target/streams-app-1.0.0.jar com.bank.streams.FraudDetectionApp

# Step 5: Register the sink connector
curl -X POST http://localhost:8083/connectors \
  -H "Content-Type: application/json" \
  -d @connect-configs/elasticsearch-sink.json

# Step 6: Insert test transactions in PostgreSQL
psql -h localhost -U bankadmin -d nationalbank -c "
INSERT INTO transactions (account_id, type, amount, description)
VALUES
  ('SAV2001', 'DEBIT', 750000, 'rtgs to vendor account'),
  ('SAV2002', 'CREDIT', 25000, 'salary credit'),
  ('CUR3001', 'DEBIT', 1200000, 'loan disbursement');"

# Step 7: Check fraud-alerts in Elasticsearch
curl http://localhost:9200/fraud-alerts/_search?pretty
TEXT
Expected result in Elasticsearch:
Hit 1: SAV2001|DEBIT|750000|SUSPICIOUS: RTGS TO VENDOR ACCOUNT|...
Hit 2: CUR3001|DEBIT|1200000|SUSPICIOUS: LOAN DISBURSEMENT|...

SAV2002 (25,000 rupees) does not appear because it is below the 5,00,000 threshold.

What are common mistakes when choosing between Kafka Streams and Kafka Connect?

Wrong: using Kafka Connect for business logic

JAVA
package com.bank.connect;

public class WrongConnectForLogic {
    public static void main(String[] args) {
        System.out.println("WRONG: Trying to add business logic in Kafka Connect");
        System.out.println();
        System.out.println("  Developer writes a Single Message Transform (SMT) to:");
        System.out.println("    - Filter transactions above 5,00,000");
        System.out.println("    - Join transaction with customer data");
        System.out.println("    - Aggregate daily totals");
        System.out.println();
        System.out.println("  Problems:");
        System.out.println("    1. SMTs are stateless, one record at a time");
        System.out.println("    2. Cannot join two topics in Connect");
        System.out.println("    3. Cannot aggregate or window in Connect");
        System.out.println("    4. Complex SMTs are hard to test and debug");
    }
}
TEXT
Expected output:
WRONG: Trying to add business logic in Kafka Connect

  Developer writes a Single Message Transform (SMT) to:
    - Filter transactions above 5,00,000
    - Join transaction with customer data
    - Aggregate daily totals

  Problems:
    1. SMTs are stateless, one record at a time
    2. Cannot join two topics in Connect
    3. Cannot aggregate or window in Connect
    4. Complex SMTs are hard to test and debug

Correct: use Connect for moving data, Streams for logic

JAVA
package com.bank.streams;

public class CorrectConnectAndStreams {
    public static void main(String[] args) {
        System.out.println("CORRECT: Each tool does what it is designed for");
        System.out.println();
        System.out.println("  Kafka Connect (source):");
        System.out.println("    - Reads transactions from PostgreSQL");
        System.out.println("    - Writes raw events to Kafka topic");
        System.out.println("    - No business logic, just data movement");
        System.out.println();
        System.out.println("  Kafka Streams:");
        System.out.println("    - Reads from Kafka topic");
        System.out.println("    - Filters, joins, aggregates");
        System.out.println("    - Writes processed results to another Kafka topic");
        System.out.println();
        System.out.println("  Kafka Connect (sink):");
        System.out.println("    - Reads processed results from Kafka topic");
        System.out.println("    - Writes to Elasticsearch for dashboards");
        System.out.println("    - No business logic, just data movement");
    }
}
TEXT
Expected output:
CORRECT: Each tool does what it is designed for

  Kafka Connect (source):
    - Reads transactions from PostgreSQL
    - Writes raw events to Kafka topic
    - No business logic, just data movement

  Kafka Streams:
    - Reads from Kafka topic
    - Filters, joins, aggregates
    - Writes processed results to another Kafka topic

  Kafka Connect (sink):
    - Reads processed results from Kafka topic
    - Writes to Elasticsearch for dashboards
    - No business logic, just data movement

What about Single Message Transforms (SMTs) in Kafka Connect?

Kafka Connect supports lightweight transformations called SMTs. They can rename fields, mask sensitive data, add timestamps, or route records. But SMTs are stateless and process one record at a time. They are not a replacement for Kafka Streams.

Use SMTs for simple field-level changes. Use Kafka Streams for anything stateful, multi-record, or involving business rules.

JAVA
package com.bank.connect;

public class SmtVsStreams {
    public static void main(String[] args) {
        System.out.println("SMTs (Kafka Connect) can do:");
        System.out.println("  - Rename a field: account_id -> accountId");
        System.out.println("  - Mask a field: card_number -> ****1234");
        System.out.println("  - Add a timestamp field");
        System.out.println("  - Route to different topics based on a field value");
        System.out.println("  - Drop a field from the record");
        System.out.println();
        System.out.println("SMTs cannot do:");
        System.out.println("  - Join two topics");
        System.out.println("  - Aggregate or count records");
        System.out.println("  - Window-based calculations");
        System.out.println("  - Maintain state across records");
        System.out.println("  - Complex filtering with external lookups");
        System.out.println();
        System.out.println("Rule: If it needs state or multiple records, use Kafka Streams.");
    }
}
TEXT
Expected output:
SMTs (Kafka Connect) can do:
  - Rename a field: account_id -> accountId
  - Mask a field: card_number -> ****1234
  - Add a timestamp field
  - Route to different topics based on a field value
  - Drop a field from the record

SMTs cannot do:
  - Join two topics
  - Aggregate or count records
  - Window-based calculations
  - Maintain state across records
  - Complex filtering with external lookups

Rule: If it needs state or multiple records, use Kafka Streams.

What should a senior or principal engineer know?

At the senior level, you need to design pipelines that use Connect and Streams together correctly. The common architecture is: Connect source brings data in, Streams processes it, Connect sink pushes it out.

At the principal level, consider:

  • Connector task parallelism: a source connector can run multiple tasks, each reading from a different table or partition. Match task count to your throughput need.
  • Schema evolution: when using Connect with Avro and Schema Registry, schema changes in the database propagate through the pipeline. Kafka Streams must handle both old and new schemas. Use a schema-aware Serde.
  • Backpressure: if Kafka Streams cannot keep up, records accumulate in the topic. Consumer lag increases. Monitor records-lag-max and scale by adding instances.
  • Connect worker sizing: in distributed mode, Connect workers share tasks. If a worker dies, its tasks are rebalanced to survivors. Plan for N+1 workers.
  • Exactly-once across the pipeline: Kafka Streams supports exactly_once_v2. For Connect, exactly-once depends on the connector. Debezium source is at-least-once; the sink connector must be idempotent.
  • Dead letter queue in Connect: configure errors.deadletterqueue.topic.name so bad records do not block the pipeline.
JAVA
package com.bank.streams;

public class PrincipalKnowledge {
    public static void main(String[] args) {
        System.out.println("Principal-level pipeline design:");
        System.out.println("  1. Connect source -> Kafka topic (raw data)");
        System.out.println("  2. Kafka Streams -> Kafka topic (processed data)");
        System.out.println("  3. Connect sink -> external store (search, analytics)");
        System.out.println();
        System.out.println("Monitoring checklist:");
        System.out.println("  - Connect: connector-status, task-status, dead-letter-queue-size");
        System.out.println("  - Streams: consumer-lag, state-store-size, stream-thread-state");
        System.out.println("  - Pipeline: end-to-end latency from DB change to Elasticsearch");
        System.out.println();
        System.out.println("Schema evolution:");
        System.out.println("  - Use Avro + Schema Registry for all topics");
        System.out.println("  - Forward compatibility: new fields have defaults");
        System.out.println("  - Streams app uses GenericAvroSerde or SpecificAvroSerde");
    }
}
TEXT
Expected output:
Principal-level pipeline design:
  1. Connect source -> Kafka topic (raw data)
  2. Kafka Streams -> Kafka topic (processed data)
  3. Connect sink -> external store (search, analytics)

Monitoring checklist:
  - Connect: connector-status, task-status, dead-letter-queue-size
  - Streams: consumer-lag, state-store-size, stream-thread-state
  - Pipeline: end-to-end latency from DB change to Elasticsearch

Schema evolution:
  - Use Avro + Schema Registry for all topics
  - Forward compatibility: new fields have defaults
  - Streams app uses GenericAvroSerde or SpecificAvroSerde

What are common interview follow-up questions?

  • Can Kafka Connect do transformations? (Yes, but only stateless Single Message Transforms. For stateful processing, use Kafka Streams.)
  • Can Kafka Streams read from a database directly? (No. Kafka Streams reads only from Kafka topics. Use Kafka Connect to ingest database data into Kafka first.)
  • What is the difference between standalone and distributed mode in Kafka Connect? (Standalone runs one worker process, good for development. Distributed runs a cluster of workers with automatic task balancing, required for production.)
  • Can you use both Kafka Streams and Kafka Connect in the same application? (They are separate concerns. Connect runs as its own cluster. Streams runs inside your Java application. They communicate through Kafka topics.)
  • What is Debezium? (An open-source Kafka Connect source connector for change data capture. It reads the database transaction log and streams every change to Kafka.)
  • How do you handle schema changes in a Connect-Streams pipeline? (Use Avro with Schema Registry. Connect publishes schemas automatically. Streams uses a schema-aware Serde to deserialize both old and new formats.)

Visual Overview

DIAGRAM
 Complete Banking Pipeline
 ==========================

 ┌──────────────┐
 │ PostgreSQL    │
 │ transactions  │
 │ table         │
 └──────┬───────┘
        │ Debezium CDC
        │ (Kafka Connect Source)
        ▼
 ┌──────────────────────┐
 │ Kafka Topic           │
 │ nbi.public.transactions│
 └──────┬───────────────┘
        │
        │ Kafka Streams
        ▼
 ┌──────────────────────┐
 │ Filter + Transform   │
 │ amount > 500,000     │
 │ uppercase description│
 └──────┬───────────────┘
        │
        ▼
 ┌──────────────────────┐
 │ Kafka Topic           │
 │ fraud-alerts          │
 └──────┬───────────────┘
        │ Elasticsearch Sink
        │ (Kafka Connect Sink)
        ▼
 ┌──────────────────────┐
 │ Elasticsearch         │
 │ fraud-alerts index    │
 │ (search + dashboards) │
 └──────────────────────┘


 Connect vs Streams
 ===================

 Connect = Move data (DB <-> Kafka)
 Streams = Process data (Kafka -> Kafka)

 ┌────────┐    ┌───────┐    ┌────────┐    ┌───────┐    ┌────────┐
 │External│───>│Connect│───>│ Kafka  │───>│Streams│───>│ Kafka  │
 │System  │    │Source │    │ Topic  │    │  App  │    │ Topic  │
 └────────┘    └───────┘    └────────┘    └───────┘    └───┬────┘
                                                          │
                                                          ▼
                                                     ┌───────┐
                                                     │Connect│
                                                     │ Sink  │
                                                     └───┬───┘
                                                         │
                                                         ▼
                                                    ┌────────┐
                                                    │External│
                                                    │System  │
                                                    └────────┘

Key Takeaway

Kafka Streams and Kafka Connect solve different problems. Kafka Connect moves data between Kafka and external systems. Kafka Streams processes data that is already inside Kafka. They are complementary, not competing.

In our National Bank of India fraud detection pipeline, Kafka Connect (Debezium) captured every transaction from the PostgreSQL database and published it to a Kafka topic. Kafka Streams read those transactions, filtered amounts above five lakh rupees, flagged them as suspicious, and wrote the results to a fraud-alerts topic. Then another Kafka Connect sink connector picked up the fraud alerts and indexed them in Elasticsearch for the compliance team's dashboard.

The key rule is: Connect for data movement, Streams for business logic. If you need to filter, join, aggregate, or window, that is Kafka Streams. If you need to read from a database, write to a search engine, or push data to a data lake, that is Kafka Connect. If you find yourself writing complex Single Message Transforms in Connect, move that logic to a Kafka Streams application.

At the principal level, design the pipeline so each component does what it is best at. Monitor connector status, consumer lag, and end-to-end latency. Use Avro with Schema Registry so schema changes in the database flow through the entire pipeline without breaking downstream consumers.

Remember one sentence: Kafka Connect moves data in and out of Kafka, Kafka Streams processes data inside Kafka, and a production pipeline uses both.

---

Share
Use ← → arrow keys to navigate