What is the Kafka Connect architecture, and what is standalone vs distributed mode? | Kafka & Messaging

Definition

Kafka Connect has four building blocks: connectors define the job, tasks move the data, workers host the tasks, and converters handle the wire format, running in either standalone mode for development or distributed mode for production fault tolerance.

Kafka Connect is a framework for streaming data between Kafka and external systems without writing custom producer or consumer code.

Worker – a JVM process that hosts and runs connector tasks, either as a standalone process or as part of a distributed cluster.
Connector – a configuration-level definition that specifies which external system to connect and how to divide the job into parallel tasks.
Task – the unit that actually moves data, reading from an external system (source task) or writing to one (sink task).
Converter – the component that serializes and deserializes record keys and values in a format such as JSON, Avro, or Protobuf.
Standalone mode – a single-worker deployment using a local properties file, suitable for development and testing.
Distributed mode – a multi-worker cluster that stores state in internal Kafka topics and is managed entirely through a REST API.

Think of Kafka Connect at National Bank of India as a courier logistics system. A Connector is the contract that says which branch to pick up transaction records from and how many delivery vans to dispatch. Each Task is one delivery van that actually loads and transports the records. Workers are the depots where vans are parked and dispatched from. Converters are the packing standards that decide whether records are wrapped in plain paper (JSON) or sealed containers (Avro). Standalone mode is a single-depot trial run for testing. Distributed mode is a full fleet across multiple depots with automatic rerouting if one depot goes offline.

Understanding the Concepts

What is Kafka Connect?

Kafka Connect is a framework built into the Apache Kafka ecosystem that moves data between Kafka topics and external systems such as databases, file systems, search indexes, and cloud storage. Instead of writing custom producer and consumer code for every integration, you configure a connector and Connect handles the data pipeline.

At National Bank of India, Rajesh uses Kafka Connect to stream transaction records from the core banking database into Kafka for downstream analytics, and to sink processed events from Kafka into the data warehouse. No custom glue code is needed.

JAVA

package com.bank.connect;

// Conceptual overview: Kafka Connect removes the need for custom code
// Source: Database -> Kafka Connect Source Task -> Kafka Topic
// Sink:   Kafka Topic -> Kafka Connect Sink Task -> Data Warehouse

// Instead of writing this manually:
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.math.BigDecimal;
import java.util.Properties;

public class ManualIngestion {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "kafka-1:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            // Manual: poll DB, build record, produce
            String accountId = "SAV2001";
            BigDecimal amount = new BigDecimal("50000.00");
            producer.send(new ProducerRecord<>("bank.transactions",
                accountId, "{\"account\":\"" + accountId + "\",\"amount\":" + amount + "}"));
        }
        // Kafka Connect replaces all of this with a JSON configuration
    }
}

TEXT

Expected output:
With Kafka Connect, no manual producer code is needed.
You submit a connector configuration and Connect handles everything.

Why does Kafka Connect exist?

Before Kafka Connect, every team wrote its own producer or consumer to bridge Kafka with external systems. This led to duplicated code, inconsistent error handling, and no standard way to track offsets or recover from failures.

Kafka Connect solves these problems by providing:

A standard plugin interface so connector authors write the integration logic once and everyone reuses it.
Automatic offset tracking so the framework knows which records have been processed.
Fault tolerance in distributed mode so tasks are reassigned when a worker fails.
Converter separation so the serialization format is independent of the connector logic.

JAVA

package com.bank.connect;

// Without Connect: every team duplicates offset tracking
// Team A: polls transactions table, tracks last_id in a file
// Team B: polls loans table, tracks last_id in another file
// Team C: polls accounts table, same pattern again

// With Connect: one JDBC source connector handles all three tables
// Offset tracking is automatic and stored in Kafka
// Fault tolerance is built in

public class WhyConnectExists {
    public static void main(String[] args) {
        System.out.println("Problem: 3 teams, 3 duplicate ingestion scripts");
        System.out.println("Solution: 1 JDBC Source Connector, 3 table configs");
        System.out.println("Benefit: automatic offset tracking, fault tolerance, no custom code");
    }
}

TEXT

Expected output:
Problem: 3 teams, 3 duplicate ingestion scripts
Solution: 1 JDBC Source Connector, 3 table configs
Benefit: automatic offset tracking, fault tolerance, no custom code

What are the four building blocks of the architecture?

The architecture has four components that each handle one responsibility.

Workers are JVM processes. They host connector instances and tasks, expose the REST API in distributed mode, and coordinate with other workers for task assignment. A worker does not move data itself; it runs the tasks that do.

Connectors are configuration-level definitions. When you submit a connector config, the connector class decides how to split the job into tasks. A JDBC source connector might create one task per table. A sink connector might create one task per set of Kafka partitions. The connector itself copies no bytes.

Tasks are the units that move data. A source task reads from an external system and produces records to Kafka. A sink task reads records from Kafka and writes them to an external system. Tasks run in parallel, and the maximum count is set by the tasks.max configuration.

Converters sit between tasks and Kafka. A source task produces records as internal Connect data structures, and the converter serializes them to bytes before they hit Kafka. A sink task's converter deserializes bytes back. Key and value converters are configured independently.

JAVA

package com.bank.connect;

import org.apache.kafka.connect.source.SourceTask;
import org.apache.kafka.connect.source.SourceRecord;
import java.util.List;
import java.util.Map;
import java.util.Collections;

/**
 * Simplified source task for National Bank of India transactions.
 * This shows the Task building block: the unit that actually moves data.
 */
public class BankTransactionSourceTask extends SourceTask {

    private String tableName;
    private long lastOffset;

    @Override
    public String version() {
        return "1.0.0";
    }

    @Override
    public void start(Map<String, String> props) {
        this.tableName = props.get("table.name"); // e.g., "transactions"
        this.lastOffset = 0L;
    }

    @Override
    public List<SourceRecord> poll() throws InterruptedException {
        // Task reads from the external system (database)
        // and produces SourceRecords for Kafka
        Map<String, String> sourcePartition = Collections.singletonMap("table", tableName);
        Map<String, Long> sourceOffset = Collections.singletonMap("position", lastOffset);

        SourceRecord record = new SourceRecord(
            sourcePartition, sourceOffset,
            "bank.transactions",  // target Kafka topic
            null,                  // partition (null = default partitioner)
            org.apache.kafka.connect.data.Schema.STRING_SCHEMA,
            "SAV2001",            // key: account id
            org.apache.kafka.connect.data.Schema.STRING_SCHEMA,
            "{\"account\":\"SAV2001\",\"amount\":50000.00}" // value
        );
        lastOffset++;
        return Collections.singletonList(record);
    }

    @Override
    public void stop() {
        // Clean up database connections
    }
}

TEXT

Expected output:
The task polls the database, creates SourceRecords, and the
Connect runtime produces them to the configured Kafka topic.
The converter serializes the record before it reaches Kafka.

How does standalone mode work?

Standalone mode runs a single worker process on one machine. Configuration is stored in local properties files, not in Kafka. You start it by passing a worker properties file and one or more connector properties files to the connect-standalone.sh script.

Standalone mode is suitable for development, testing, and single-machine edge deployments. It has no fault tolerance: if the process crashes, the pipeline stops.

TEXT

# File: standalone-worker.properties
bootstrap.servers=kafka-1:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
offset.storage.file.filename=/tmp/connect-offsets.dat

TEXT

# File: bank-source-connector.properties
name=bank-transaction-source
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:postgresql://localhost:5432/bankdb
connection.user=rajesh
connection.password=secret
table.whitelist=transactions,accounts
mode=incrementing
incrementing.column.name=txn_id
topic.prefix=bank.

BASH

# Start standalone worker with one connector
bin/connect-standalone.sh \
  config/standalone-worker.properties \
  config/bank-source-connector.properties

TEXT

Expected output:
[INFO] WorkerInfo values:
[INFO] Kafka Connect started
[INFO] Starting connector bank-transaction-source
[INFO] Source task started, polling table: transactions

How does distributed mode work?

Distributed mode runs multiple worker processes that form a cluster. They coordinate through Kafka internal topics that store connector configurations, task offsets, and connector status. Connectors are managed entirely through a REST API, not local files.

When a worker joins or leaves the cluster, a rebalance distributes tasks evenly across the surviving workers. This is how distributed mode provides fault tolerance and horizontal scaling.

TEXT

# File: distributed-worker.properties
bootstrap.servers=kafka-1:9092,kafka-2:9092,kafka-3:9092
group.id=bank-connect-cluster
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://schema-registry:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://schema-registry:8081

# Internal topics for storing state
config.storage.topic=bank-connect-configs
config.storage.replication.factor=3
offset.storage.topic=bank-connect-offsets
offset.storage.replication.factor=3
status.storage.topic=bank-connect-status
status.storage.replication.factor=3

rest.port=8083

BASH

# Start distributed worker on each machine
bin/connect-distributed.sh config/distributed-worker.properties

# Create a connector via REST API
curl -X POST http://localhost:8083/connectors \
  -H "Content-Type: application/json" \
  -d '{
    "name": "bank-transaction-source",
    "config": {
      "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
      "tasks.max": "3",
      "connection.url": "jdbc:postgresql://bankdb:5432/nbi",
      "table.whitelist": "transactions,accounts,loans",
      "mode": "incrementing",
      "incrementing.column.name": "txn_id",
      "topic.prefix": "bank."
    }
  }'

# Check connector status
curl http://localhost:8083/connectors/bank-transaction-source/status

TEXT

Expected output:
{"name":"bank-transaction-source","connector":{"state":"RUNNING","worker_id":"worker-1:8083"},
 "tasks":[{"id":0,"state":"RUNNING","worker_id":"worker-1:8083"},
          {"id":1,"state":"RUNNING","worker_id":"worker-2:8083"},
          {"id":2,"state":"RUNNING","worker_id":"worker-3:8083"}]}

How is standalone different from distributed?

The two modes differ in configuration storage, management interface, fault tolerance, and scaling.

| Feature | Standalone Mode | Distributed Mode |
|---|---|---|
| Workers | Single JVM process | Multiple coordinated workers |
| Config storage | Local properties files | Internal Kafka topics |
| Offset storage | Local file on disk | Internal Kafka topic |
| Management | Command-line arguments | REST API on port 8083 |
| Fault tolerance | None: crash stops pipeline | Tasks rebalanced to surviving workers |
| Scaling | Vertical only (one machine) | Horizontal: add more workers |
| Use case | Development and testing | Production deployments |
| State durability | Lost if disk fails | Replicated in Kafka |

JAVA

package com.bank.connect;

public class ModeComparison {
    public static void main(String[] args) {
        System.out.println("=== Standalone Mode ===");
        System.out.println("Config: local .properties file");
        System.out.println("Offsets: /tmp/connect-offsets.dat");
        System.out.println("Fault tolerance: NONE");
        System.out.println();
        System.out.println("=== Distributed Mode ===");
        System.out.println("Config: bank-connect-configs topic (RF=3)");
        System.out.println("Offsets: bank-connect-offsets topic (RF=3)");
        System.out.println("Fault tolerance: task rebalancing across workers");
    }
}

TEXT

Expected output:
=== Standalone Mode ===
Config: local .properties file
Offsets: /tmp/connect-offsets.dat
Fault tolerance: NONE

=== Distributed Mode ===
Config: bank-connect-configs topic (RF=3)
Offsets: bank-connect-offsets topic (RF=3)
Fault tolerance: task rebalancing across workers

How do converters handle data serialization?

Converters sit between the task and Kafka. When a source task produces a record, the converter serializes the key and value to bytes before writing to Kafka. When a sink task consumes a record, the converter deserializes the bytes back into Connect data structures.

Key and value converters are configured independently. The converter must match across the pipeline: if the source writes Avro, the sink must read Avro.

The three common converters are:

JsonConverter produces human-readable JSON. Simple but larger on the wire.
AvroConverter produces compact binary with a schema stored in Schema Registry. Best for production.
ProtobufConverter produces compact binary using Protocol Buffer schemas.

TEXT

# JSON Converter (human-readable, no schema registry needed)
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=false

# Avro Converter (compact binary, requires schema registry)
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://schema-registry:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://schema-registry:8081

JAVA

package com.bank.connect;

public class ConverterDemo {
    public static void main(String[] args) {
        // JSON output for account SAV2001
        String jsonOutput = "{\"account\":\"SAV2001\",\"name\":\"Priya\",\"balance\":250000.00}";
        System.out.println("JSON Converter output: " + jsonOutput);
        System.out.println("JSON size: " + jsonOutput.getBytes().length + " bytes");

        // Avro output is binary, roughly half the size
        System.out.println("Avro size: ~30 bytes (compact binary + schema in registry)");
        System.out.println();
        System.out.println("Rule: source and sink converters MUST match.");
        System.out.println("If source writes Avro, sink must read Avro.");
    }
}

TEXT

Expected output:
JSON Converter output: {"account":"SAV2001","name":"Priya","balance":250000.00}
JSON size: 55 bytes
Avro size: ~30 bytes (compact binary + schema in registry)

Rule: source and sink converters MUST match.
If source writes Avro, sink must read Avro.

What are common mistakes?

The most frequent mistake is a converter mismatch between source and sink. If the source writes JSON but the sink expects Avro, every record fails deserialization.

Wrong: mismatched converters across a pipeline.

TEXT

# Source connector writes JSON
value.converter=org.apache.kafka.connect.json.JsonConverter

# Sink connector expects Avro -- MISMATCH!
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://schema-registry:8081

TEXT

Error output:
org.apache.kafka.connect.errors.DataException:
  Failed to deserialize data for topic bank.transactions to Avro:
  Unknown magic byte!

Correct: matching converters on both sides.

TEXT

# Source connector writes Avro
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://schema-registry:8081

# Sink connector reads Avro -- MATCH
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://schema-registry:8081

Another common mistake is setting tasks.max higher than the natural parallelism. If you have 2 source tables but set tasks.max=10, only 2 tasks will have work and the other 8 sit idle.

Wrong: tasks.max exceeds the natural parallelism.

TEXT

# Only 2 tables but 10 tasks requested
table.whitelist=transactions,accounts
tasks.max=10
# Result: 2 active tasks, 8 idle tasks wasting resources

Correct: tasks.max matches the workload.

TEXT

# 2 tables, 2 tasks -- each task handles one table
table.whitelist=transactions,accounts
tasks.max=2

What about fault tolerance in distributed mode?

In distributed mode, when a worker crashes, the cluster detects the failure and triggers a rebalance. The surviving workers take over the tasks that were running on the failed worker. No data is lost because source offsets are stored in the internal offsets topic and sink offsets are tracked by Kafka consumer group commits.

At National Bank of India, Pankaj runs a three-worker Connect cluster. When worker-2 crashes during a deployment, worker-1 and worker-3 each pick up one of worker-2's tasks within seconds. The transaction ingestion pipeline continues without manual intervention.

JAVA

package com.bank.connect;

public class FaultToleranceDemo {
    public static void main(String[] args) {
        System.out.println("=== Before failure ===");
        System.out.println("Worker-1: Task-0 (transactions table)");
        System.out.println("Worker-2: Task-1 (accounts table)");
        System.out.println("Worker-3: Task-2 (loans table)");
        System.out.println();
        System.out.println("=== Worker-2 crashes ===");
        System.out.println("Rebalance triggered...");
        System.out.println();
        System.out.println("=== After rebalance ===");
        System.out.println("Worker-1: Task-0 (transactions), Task-1 (accounts)");
        System.out.println("Worker-3: Task-2 (loans)");
        System.out.println("No data lost. Offsets stored in Kafka.");
    }
}

TEXT

Expected output:
=== Before failure ===
Worker-1: Task-0 (transactions table)
Worker-2: Task-1 (accounts table)
Worker-3: Task-2 (loans table)

=== Worker-2 crashes ===
Rebalance triggered...

=== After rebalance ===
Worker-1: Task-0 (transactions), Task-1 (accounts)
Worker-3: Task-2 (loans)
No data lost. Offsets stored in Kafka.

What should a senior engineer know?

A senior engineer should understand these operational nuances:

All workers in a distributed cluster must share the same group.id to form one cluster. Different group.id values create separate clusters.
Internal config, offset, and status topics must be replicated (typically replication.factor=3) for durability. Losing these topics means losing connector state.
The REST API should be secured with TLS and authentication because it can create, delete, and modify connectors across the entire cluster.
Connector configurations should be version-controlled and applied through CI/CD automation, not ad-hoc curl commands, so the cluster state is reproducible.
Single Message Transforms (SMTs) let you modify records in the pipeline without writing custom code, for example masking Priya's account number before it reaches the data warehouse.

JAVA

package com.bank.connect;

public class SeniorNuances {
    public static void main(String[] args) {
        System.out.println("1. group.id must match across all workers in one cluster");
        System.out.println("2. Internal topics replication factor = 3 (minimum)");
        System.out.println("3. Secure REST API with TLS + authentication");
        System.out.println("4. Version-control connector configs in Git");
        System.out.println("5. Use SMTs for field masking, routing, timestamping");
        System.out.println("6. tasks.max > natural parallelism = idle tasks");
        System.out.println("7. Monitor: task state, record rate, consumer lag");
    }
}

TEXT

Expected output:
1. group.id must match across all workers in one cluster
2. Internal topics replication factor = 3 (minimum)
3. Secure REST API with TLS + authentication
4. Version-control connector configs in Git
5. Use SMTs for field masking, routing, timestamping
6. tasks.max > natural parallelism = idle tasks
7. Monitor: task state, record rate, consumer lag

Visual Overview

DIAGRAM

  ┌─────────────────────────────────────────────────────┐
  │              Kafka Connect Architecture               │
  │                                                       │
  │  ┌─────────────┐   ┌─────────────┐   ┌────────────┐ │
  │  │  Worker 1    │   │  Worker 2    │   │  Worker 3  │ │
  │  │  ┌────────┐  │   │  ┌────────┐  │   │  ┌───────┐│ │
  │  │  │ Task 0 │  │   │  │ Task 1 │  │   │  │Task 2 ││ │
  │  │  └────────┘  │   │  └────────┘  │   │  └───────┘│ │
  │  │  REST :8083  │   │  REST :8083  │   │  REST:8083│ │
  │  └──────┬───────┘   └──────┬───────┘   └─────┬─────┘ │
  │         │                  │                  │       │
  │         └──────────────────┼──────────────────┘       │
  │                            │                          │
  │                    ┌───────▼────────┐                 │
  │                    │  Kafka Cluster  │                 │
  │                    │  (data topics + │                 │
  │                    │  internal topics)│                │
  │                    └─────────────────┘                 │
  └─────────────────────────────────────────────────────┘

  Standalone vs Distributed:

  Standalone:                    Distributed:
  ┌──────────────────┐           ┌──────────┐ ┌──────────┐
  │  Single Worker   │           │ Worker 1 │ │ Worker 2 │
  │  ┌──────┐        │           │ ┌──────┐ │ │ ┌──────┐ │
  │  │Task 0│        │           │ │Task 0│ │ │ │Task 1│ │
  │  └──────┘        │           │ └──────┘ │ │ └──────┘ │
  │  Config: .props  │           │ Config:  │ │ Config:  │
  │  Offset: file    │           │ Kafka    │ │ Kafka    │
  └──────────────────┘           └──────────┘ └──────────┘
  No fault tolerance.            Fault tolerant.
  Dev/testing only.              Production ready.

Key Takeaway

Kafka Connect is a framework for streaming data between Kafka and external systems without writing custom producer or consumer code. Its architecture has four building blocks: workers are JVM processes that host tasks, connectors define what to connect and how to split the work, tasks do the actual data movement, and converters handle serialization in formats like JSON or Avro. At National Bank of India, Rajesh configures a JDBC source connector to stream transactions from the core banking database into Kafka, and the framework handles offset tracking, parallelism, and recovery automatically.

The mode you choose is an operational decision. Standalone mode runs a single worker from local properties files, which is quick for development but offers no fault tolerance. Distributed mode runs a cluster of workers that coordinate through internal Kafka topics and are managed via a REST API. When a worker crashes in distributed mode, the surviving workers pick up its tasks through rebalancing, so the pipeline continues without manual intervention. Pankaj runs National Bank of India's production Connect cluster in distributed mode with three workers and replication factor three on all internal topics.

The senior nuances that matter in production are that converters must match across the pipeline or the sink cannot read what the source wrote, that tasks.max beyond the natural parallelism just creates idle tasks, that the REST API must be secured because it can modify connectors across the entire cluster, and that connector configurations belong in version control so the cluster state is reproducible.

Remember one sentence: connectors plan the work, tasks do the work, workers host the tasks, and distributed mode gives you fault tolerance by storing everything in Kafka and rebalancing tasks across surviving workers.

---