Definition
Kafka Connect is a framework and runtime for streaming data between Kafka and external systems using configuration.
- Source Connector – pulls data from an external system (database, file, API) into Kafka topics; e.g., JDBC Source reads tables.
- Sink Connector – pushes data from Kafka topics out to an external system (S3, database, Elasticsearch); e.g., S3 Sink writes files.
- Connect runtime – manages connector lifecycle, scaling into tasks, fault tolerance, and automatic offset tracking.
- Converter – decides the wire format for records: JSON, Avro, or Protobuf.
- Single Message Transform (SMT) – lightweight, per-record transformations applied in the connector pipeline without code.
Kafka Connect is like hiring a specialized courier service at National Bank of India instead of training your own messengers. A source courier knows how to walk into the account database, copy out new rows, and drop them into the Kafka sorting office. A sink courier takes sorted messages out of the Kafka office and files them neatly into cloud storage. You never teach the couriers how to drive; you just fill out a form saying which building to visit and what to carry, and the courier company handles vehicles, backups, and remembering where it left off.
Understanding the Concepts
What is Kafka Connect?
Kafka Connect is a framework that packages the repetitive plumbing of moving data between Kafka and other systems. You pick a connector that knows a specific system, give it a JSON configuration, and submit it to the Connect runtime. There is no producer or consumer code to write, build, or deploy.
# Start Connect in distributed mode
connect-distributed.sh /etc/kafka/connect-distributed.properties
# The runtime manages connectors, tasks, offsets, and fault tolerance
How does a source connector work?
A source connector imports data into Kafka. The JDBC Source Connector is the classic example: you configure the database connection, a table or query, and a mode such as polling by incrementing ID or timestamp. Connect reads new rows and produces them as records into a Kafka topic.
{
"name": "accounts-source",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"connection.url": "jdbc:postgresql://db-1:5432/banking",
"connection.user": "kafka_connect",
"connection.password": "${vault:db-password}",
"table.whitelist": "accounts",
"mode": "timestamp+incrementing",
"timestamp.column.name": "updated_at",
"incrementing.column.name": "id",
"topic.prefix": "db.",
"poll.interval.ms": "5000",
"tasks.max": "3"
}
}
# Submit the connector
curl -X POST http://connect-1:8083/connectors \
-H 'Content-Type: application/json' \
-d @accounts-source.json
# Result: rows from 'accounts' table appear in topic 'db.accounts'
How does a sink connector work?
A sink connector exports data from Kafka to an external system. The S3 Sink Connector consumes records and writes them as files to object storage, committing Kafka consumer offsets only after the files are safely written.
{
"name": "accounts-s3-sink",
"config": {
"connector.class": "io.confluent.connect.s3.S3SinkConnector",
"topics": "db.accounts",
"s3.bucket.name": "nbi-data-lake",
"s3.region": "ap-south-1",
"flush.size": "1000",
"storage.class": "io.confluent.connect.s3.storage.S3Storage",
"format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
"tasks.max": "3"
}
}
curl -X POST http://connect-1:8083/connectors \
-H 'Content-Type: application/json' \
-d @accounts-s3-sink.json
# Result: db.accounts topic records written as Parquet files to S3
How does Connect handle scaling and fault tolerance?
Connect divides a connector's work into tasks and spreads them across worker nodes. If a task or worker fails, the runtime restarts the task on a healthy worker. Offset tracking is automatic.
Distributed mode with 3 workers:
Worker-1: accounts-source task-0 (tables A-D)
Worker-2: accounts-source task-1 (tables E-H)
Worker-3: accounts-source task-2 (tables I-L)
Worker-2 crashes:
Worker-1: task-0 + task-1 (rebalanced)
Worker-3: task-2
Task-1 resumes from last source offset -- no data lost
What are converters and why do they matter?
Converters decide the on-the-wire format for records. Both source and sink must agree on the format.
# connect-distributed.properties
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://schema-registry:8081
# JSON converter (simpler but no schema enforcement):
# value.converter=org.apache.kafka.connect.json.JsonConverter
What are Single Message Transforms?
SMTs provide lightweight, per-record transformations in the connector pipeline. Use them for small reshaping; heavy logic belongs in a stream-processing application.
{
"transforms": "addPrefix,maskSsn",
"transforms.addPrefix.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.addPrefix.regex": "(.*)",
"transforms.addPrefix.replacement": "prod.$1",
"transforms.maskSsn.type": "org.apache.kafka.connect.transforms.MaskField$Value",
"transforms.maskSsn.fields": "ssn"
}
# Input topic: db.accounts
# After SMT: prod.db.accounts (prefix added, ssn field masked)
What are common mistakes with Kafka Connect?
The most common mistake is reinventing a JDBC source as a brittle custom poller when a connector already exists. The second is not monitoring connector status.
# WRONG: hand-written poller with manual offset tracking
# Loses its place on restart, misses deletes, adds query load
# CORRECT: use the JDBC Source Connector
# Handles offset tracking, restart resumption, and scaling automatically
# Monitor connector status
curl http://connect-1:8083/connectors/accounts-source/status | jq
# Look for: "state": "RUNNING" on connector and all tasks
What should a senior engineer know?
Connect is ideal when a pre-built connector exists and the transformation is light. For heavy stateful processing, use a stream-processing application. Connect is at-least-once by default, so sink targets should be idempotent. Pair Avro with Schema Registry for production-grade schema evolution.
# Production checklist:
# 1. Distributed mode with 3+ workers for fault tolerance
# 2. Avro + Schema Registry for schema evolution
# 3. Monitor connector and task status via REST API
# 4. Sink targets must be idempotent (upsert by primary key)
# 5. Use SMTs for light transforms only
Visual Overview
Kafka Connect Architecture
Source Direction (external -> Kafka):
┌──────────────┐ ┌─────────────────┐ ┌────────────┐
│ PostgreSQL DB │───→│ JDBC Source │───→│ Kafka Topic│
│ accounts table│ │ Connector │ │ db.accounts│
└──────────────┘ │ (polls new rows) │ └────────────┘
│ tracks source │
│ offsets in Kafka │
└─────────────────┘
Sink Direction (Kafka -> external):
┌────────────┐ ┌─────────────────┐ ┌──────────────┐
│ Kafka Topic│───→│ S3 Sink │───→│ S3 Bucket │
│ db.accounts│ │ Connector │ │ Parquet files│
└────────────┘ │ (writes files, │ └──────────────┘
│ commits offsets) │
└─────────────────┘
Connect Runtime:
┌────────────────────────────────────────────┐
│ Worker-1 Worker-2 Worker-3 │
│ task-0 task-1 task-2 │
│ │
│ Handles: scaling, fault tolerance, │
│ offset tracking, REST API │
└────────────────────────────────────────────┘
Key Takeaway
Kafka Connect is a framework for streaming data between Kafka and external systems using configuration instead of custom code. A source connector pulls data from an external system into Kafka topics, and a sink connector pushes data from Kafka topics out to an external system. The runtime handles scaling by splitting work into tasks across workers, fault tolerance by restarting failed tasks, and offset tracking automatically.
At National Bank of India, a JDBC source connector streamed every account change into a Kafka topic for the analytics team, and an S3 sink archived that topic as Parquet files for compliance. Both survived restarts because Connect manages their offsets: the source resumed from the last imported row and the sink committed only after each file was written.
The senior nuances are that Connect shines when a pre-built connector exists and the transformation is light (use SMTs for small tweaks), while heavy stateful processing belongs in a stream-processing application. Converters choose the wire format, with Avro plus Schema Registry as the production standard. Connect is at-least-once by default, so sink targets should be idempotent.
Remember one sentence: Kafka Connect moves data in and out of Kafka by configuration, with the runtime handling scaling, fault tolerance, and offset tracking so you write no code.
---