What is the difference between Kafka Connect and writing your own producer/consumer? | Kafka & Messaging

Q: What does a hand-written producer/consumer look like?

With a handwritten integration you own the entire lifecycle: configure a client, build records, call send, handle futures, deal with retries, write the poll loop, deserialize, process, commit offsets, handle rebalances, manage threading and shutdown. package com.bank.kafka; import org.apache.kafka.c

Q: What does Connect handle that you must build yourself?

Connect: check all connectors and their status curl http://connect1:8083/connectors jq curl http://connect1:8083/connectors/settlementsjdbcsource/status jq

Definition

Use Kafka Connect for standard, config-only data movement with built-in scaling and offset handling, and write your own producer or consumer only when you need custom logic that no connector provides.

Kafka Connect and a custom producer/consumer are two ways to move data, with a clear trade-off between configuration and control.

Kafka Connect – config-only; the runtime handles scaling, fault tolerance, and offset tracking; 100+ pre-built connectors.
Custom producer/consumer – code you own; full control over logic, but you handle poll loops, commits, retries, threading, and monitoring.
Connector ecosystem – pre-built, tested connectors for JDBC, S3, Elasticsearch, MongoDB, Debezium CDC, and many more.
Single Message Transforms – lightweight reshaping in Connect; heavy logic needs a stream-processing application.
Decision rule – use Connect when a connector exists and transformation is light; write code when logic is bespoke or stateful.

At National Bank of India, Kafka Connect is like using a standard pre-approved courier contract: you tick a box for which building and what to carry, and the courier company handles trucks, backups, and remembering the last delivery. Writing your own producer or consumer is like hiring and training your own private driver: far more work and ongoing maintenance, but the only option when you need them to do something unusual that the standard contract does not cover, like inspecting each parcel and making a decision on the spot.

Understanding the Concepts

What does a hand-written producer/consumer look like?

With a hand-written integration you own the entire lifecycle: configure a client, build records, call send, handle futures, deal with retries, write the poll loop, deserialize, process, commit offsets, handle rebalances, manage threading and shutdown.

JAVA

package com.bank.kafka;

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;

public class CustomProducerExample {
    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.ACKS_CONFIG, "all");
        props.put(ProducerConfig.RETRIES_CONFIG, 3);

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            // You handle: record construction, error handling, retries, monitoring
            producer.send(new ProducerRecord<>("settlements", "SAV2001",
                "{\"amount\":50000}")).get();
            System.out.println("Custom producer: you own every line of plumbing");
        }
    }
}

TEXT

Custom producer: you own every line of plumbing

What does Kafka Connect look like for the same job?

Connect removes the plumbing. You submit a JSON configuration and the runtime runs it.

JSON

{
  "name": "settlements-jdbc-source",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "connection.url": "jdbc:postgresql://db-1:5432/banking",
    "connection.user": "kafka_connect",
    "table.whitelist": "settlements",
    "mode": "timestamp+incrementing",
    "timestamp.column.name": "updated_at",
    "incrementing.column.name": "id",
    "topic.prefix": "db.",
    "tasks.max": "3"
  }
}

BASH

curl -X POST http://connect-1:8083/connectors \
  -H 'Content-Type: application/json' -d @config.json
# No code. Runtime handles scaling, restarts, offset tracking.

What does Connect handle that you must build yourself?

| Concern | Kafka Connect | Custom Code |
|---|---|---|
| Scaling | Automatic (tasks across workers) | You build thread pools |
| Fault tolerance | Restart failed tasks on healthy workers | You handle restarts |
| Offset tracking | Automatic (source and consumer offsets) | You commit manually |
| Configuration | JSON config, REST API | Java properties, code |
| Monitoring | REST API status endpoint | Custom metrics |
| Connector ecosystem | 100+ pre-built | You write from scratch |

BASH

# Connect: check all connectors and their status
curl http://connect-1:8083/connectors | jq
curl http://connect-1:8083/connectors/settlements-jdbc-source/status | jq

When do you choose Kafka Connect?

Choose Connect when a suitable connector exists, the transformation is light (use SMTs), and you want operational concerns handled for you. This is the majority of data-in, data-out cases.

BASH

# Common connectors available:
# JDBC Source/Sink      -- any relational database
# S3 Sink               -- archive to object storage
# Elasticsearch Sink    -- full-text search index
# MongoDB Source/Sink   -- document store
# Debezium CDC Source   -- change data capture from DB logs
# Google BigQuery Sink  -- data warehouse loading

When do you choose a custom producer/consumer?

Choose custom code when no connector fits the system or protocol, when you need rich stateful business logic, when the processing is really an application, or when you want tight control over semantics.

JAVA

package com.bank.kafka;

import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.List;
import java.util.Properties;

public class CustomFraudScorer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "fraud-scorer");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(List.of("card.authorizations"));
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                for (ConsumerRecord<String, String> record : records) {
                    // Rich business logic: score fraud, branch on result
                    double score = scoreFraud(record.value());
                    if (score > 0.8) blockCard(record.key());
                    // No connector can express this per-record decision
                }
            }
        }
    }

    static double scoreFraud(String txn) { return 0.85; }
    static void blockCard(String accountId) {
        System.out.println("BLOCK CARD: " + accountId);
    }
}

TEXT

BLOCK CARD: SAV2001

What is the common anti-pattern?

Reinventing a JDBC source as a brittle custom poller when a connector already exists. The custom poller loses its place on restart, misses deletes, and adds query load to the database.

JAVA

package com.bank.kafka;

public class AntiPatternDemo {
    public static void main(String[] args) {
        // WRONG: custom poller that does what JDBC Source Connector already does
        System.out.println("WRONG: SELECT * FROM accounts WHERE updated_at > last_poll");
        System.out.println("Problems: misses deletes, loses offset on restart, adds DB load");

        // CORRECT: use the JDBC Source Connector
        System.out.println("CORRECT: Submit JDBC Source config to Connect runtime");
        System.out.println("Handles: offset tracking, restart, scaling, fault tolerance");
    }
}

TEXT

WRONG: SELECT * FROM accounts WHERE updated_at > last_poll
Problems: misses deletes, loses offset on restart, adds DB load
CORRECT: Submit JDBC Source config to Connect runtime
Handles: offset tracking, restart, scaling, fault tolerance

What about heavy processing?

Heavy enrichment, joins, or stateful processing belongs in a stream-processing application (Kafka Streams, Flink), with Connect handling ingest and egress at the edges.

TEXT

# Architecture:
# Connect Source -> Kafka topic -> Stream processing app -> Kafka topic -> Connect Sink
#   (data in)                        (business logic)                     (data out)

# Connect handles the edges; stream processing handles the middle

What should a senior engineer know?

Connect runs in distributed mode in production for resilience. Deploy it alongside custom apps. Connect is at-least-once by default, so sink idempotency still matters. Use Avro with Schema Registry for production-grade schema evolution.

BASH

# Decision tree:
# Is there a connector for this system? -> Yes -> Use Connect
#                                        -> No  -> Write custom code
# Does the data need stateful processing? -> Yes -> Stream processing app
#                                           -> No  -> Connect with SMTs

Visual Overview

DIAGRAM

  Decision: Kafka Connect vs Custom Code

  ┌─────────────────────────────────────────┐
  │ Standard data movement?                  │
  │ (DB -> Kafka, Kafka -> S3, CDC, etc.)   │
  │                                          │
  │ YES: Kafka Connect                       │
  │ ┌────────────────────────────────┐       │
  │ │ JSON config -> REST API        │       │
  │ │ Runtime handles:               │       │
  │ │   scaling, restarts, offsets   │       │
  │ │ 100+ pre-built connectors      │       │
  │ └────────────────────────────────┘       │
  │                                          │
  │ NO: Custom Producer/Consumer             │
  │ ┌────────────────────────────────┐       │
  │ │ Java code you own              │       │
  │ │ Full control:                  │       │
  │ │   logic, branching, state      │       │
  │ │ You handle:                    │       │
  │ │   poll loop, commits, retries  │       │
  │ └────────────────────────────────┘       │
  └─────────────────────────────────────────┘

Key Takeaway

Kafka Connect and a custom producer/consumer are two ways to move data, and the difference is a clear trade-off between configuration and control. Connect is config-only: for standard integrations you write no code, and the runtime handles scaling, fault tolerance, and offset tracking automatically. It ships with 100+ pre-built connectors so an integration that would take days to harden as custom code is often a single configuration file.

A custom producer/consumer is code you own, which means you handle the client, poll loop, commits, retries, and monitoring yourself. That is the right level when the logic is bespoke, when no connector fits, or when processing is really an application that enriches and branches on each record.

At National Bank of India, Pankaj's team replaced a brittle custom poller that kept losing its place with a JDBC source connector, making the problem disappear because Connect tracks offsets automatically. But enriching each transaction with a live fraud score and branching on the result stayed as a custom consumer, because that per-record decision is logic no connector can express.

Remember one sentence: use Connect when a connector exists and transformation is light, and write custom code only when you need bespoke logic that no connector can express.

---