What is Debezium, and how does change data capture stream database changes into Kafka? | Kafka & Messaging

Q: How do you consume Debezium events in a Spring Kafka listener?

package com.bank.kafka; import com.fasterxml.jackson.databind.JsonNode; import org.springframework.kafka.annotation.KafkaListener; import org.springframework.stereotype.Component; @Component public class AccountCdcConsumer { @KafkaListener(topics = "cdc.public.accounts", groupId = "accountsync") pub

Q: What should a senior engineer know?

Debezium is atleastonce: a restart can replay recent events, so consumers must be idempotent (keyed by primary key). The PostgreSQL replication slot must be monitored because an offline consumer causes the database to retain WAL and fill the disk. Schema changes need handling, commonly Avro with Sch

Definition

Debezium captures every insert, update, and delete by reading the database's transaction log and streams them as ordered Kafka events, with no polling and the full before-and-after of each row.

Debezium is an open-source CDC source connector for Kafka Connect that streams row-level changes from a database.

Change Data Capture (CDC) – capturing database changes by reading the transaction log rather than polling tables.
Transaction log – the database's own durability record: MySQL binlog, PostgreSQL WAL, MongoDB oplog.
Before/after images – each event carries the row's state before and after the change.
Tombstone event – a null-value record emitted after a delete, enabling log compaction to forget the deleted key.
Initial snapshot – on first connect, Debezium reads the full current table state before switching to live log streaming.

Imagine National Bank of India keeps a tamper-proof logbook (the transaction log) where the database records every single change to every account in exact order. Debezium is a dedicated reader who sits beside that logbook and, the instant a line is written, announces it over the intercom into Kafka: Rajesh's balance went from one hundred thousand to ninety-five thousand, Priya's account was created, this account was closed. It never interrupts the tellers by asking questions; it just quietly reads the official log and broadcasts each change as it lands.

Understanding the Concepts

What is change data capture and why is it better than polling?

The alternative to CDC is polling, where a connector repeatedly runs a query. Polling misses changes between polls if timestamps collide, cannot see deletes, adds query load, and has latency equal to the poll interval. CDC reads the database's own transaction log, which is the authoritative, ordered record the database writes for durability.

| Aspect | Polling (JDBC Source) | CDC (Debezium) |
|---|---|---|
| Captures deletes | No (deleted rows not in query) | Yes (log records DELETE) |
| Misses changes | Possible (timestamp collision) | No (log is complete) |
| Query load on DB | Yes (repeated SELECT) | No (reads existing log) |
| Latency | Poll interval (seconds) | Milliseconds |
| Before/after image | No | Yes |

BASH

# Polling: misses Rajesh's account deletion (row gone from table)
SELECT * FROM accounts WHERE updated_at > '2024-01-15 10:00:00';
# Returns nothing for deleted rows!

# Debezium: captures the DELETE from the WAL
# Event: {"op":"d", "before":{"id":"SAV2001","balance":0}, "after":null}

How do you configure Debezium for PostgreSQL?

Debezium runs as a Kafka Connect source connector. You submit a JSON configuration specifying the database, the logical replication slot, and the tables to capture.

JSON

{
  "name": "accounts-cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "db-1",
    "database.port": "5432",
    "database.user": "debezium",
    "database.password": "${vault:cdc-password}",
    "database.dbname": "banking",
    "database.server.name": "nbi",
    "table.include.list": "public.accounts",
    "plugin.name": "pgoutput",
    "slot.name": "debezium_accounts",
    "topic.prefix": "cdc",
    "snapshot.mode": "initial",
    "tasks.max": "1"
  }
}

BASH

curl -X POST http://connect-1:8083/connectors \
  -H 'Content-Type: application/json' -d @accounts-cdc.json

# Result: all changes to accounts table stream to topic cdc.public.accounts

What does a Debezium change event look like?

Each event carries an operation field, before and after images of the row, and source metadata.

JSON

{
  "op": "u",
  "before": {
    "id": "SAV2001",
    "name": "Rajesh",
    "balance": 100000
  },
  "after": {
    "id": "SAV2001",
    "name": "Rajesh",
    "balance": 95000
  },
  "source": {
    "db": "banking",
    "table": "accounts",
    "lsn": 12345678,
    "ts_ms": 1705312800000
  }
}

TEXT

# op="c" -> CREATE (insert)
# op="u" -> UPDATE (before + after)
# op="d" -> DELETE (before only, after is null)
# op="r" -> READ (from initial snapshot)

How does the initial snapshot work?

When Debezium first connects, it reads the current table contents as a complete snapshot so downstream consumers start with the full picture. After the snapshot completes, it switches to streaming live changes from the log.

TEXT

First connection:
1. Lock table briefly to get a consistent snapshot point
2. Read all existing rows -> emit as "r" (read) events
3. Record the log position at snapshot time
4. Switch to streaming from that log position forward
5. All subsequent changes are "c", "u", or "d" events

How do you consume Debezium events in a Spring Kafka listener?

JAVA

package com.bank.kafka;

import com.fasterxml.jackson.databind.JsonNode;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

@Component
public class AccountCdcConsumer {

    @KafkaListener(topics = "cdc.public.accounts", groupId = "account-sync")
    public void onAccountChange(JsonNode event) {
        String op = event.get("op").asText();
        switch (op) {
            case "c", "r" -> {
                JsonNode after = event.get("after");
                System.out.println("CREATE/SNAPSHOT: " + after.get("id").asText()
                    + " balance=" + after.get("balance"));
            }
            case "u" -> {
                JsonNode before = event.get("before");
                JsonNode after = event.get("after");
                System.out.println("UPDATE: " + after.get("id").asText()
                    + " " + before.get("balance") + " -> " + after.get("balance"));
            }
            case "d" -> {
                JsonNode before = event.get("before");
                System.out.println("DELETE: " + before.get("id").asText());
            }
        }
    }
}

TEXT

CREATE/SNAPSHOT: SAV2001 balance=100000
UPDATE: SAV2001 100000 -> 95000
DELETE: SAV2001

How does Debezium support the transactional outbox pattern?

Debezium reliably publishes domain events written to an outbox table in the same local transaction as the business change, solving the dual-write problem.

JAVA

package com.bank.service;

import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;

@Service
public class TransferWithOutbox {

    @Transactional
    public void transfer(String from, String to, BigDecimal amount) {
        accountRepo.debit(from, amount);
        accountRepo.credit(to, amount);
        // Same DB transaction -> atomic with business change
        outboxRepo.insert("TransferCompleted", from,
            "{\"from\":\"" + from + "\",\"to\":\"" + to
                + "\",\"amount\":" + amount + "}");
    }
}
// Debezium CDC reads the outbox table and streams events to Kafka

TEXT

DB transaction: debit SAV2001 + credit SAV3001 + outbox insert -> COMMIT
Debezium: reads outbox change -> publishes to Kafka topic

What about tombstone events and log compaction?

After emitting a delete event, Debezium sends a tombstone (null value) for the key. On a compacted topic, this tombstone causes the log compactor to eventually remove the key entirely, so the compacted topic correctly forgets deleted accounts.

TEXT

# Delete of SAV2001:
# Event 1: {"op":"d", "before":{"id":"SAV2001",...}, "after":null}
# Event 2: key=SAV2001, value=null  (tombstone)

# On a compacted topic, after delete.retention.ms:
# SAV2001 is completely removed from the log

What should a senior engineer know?

Debezium is at-least-once: a restart can replay recent events, so consumers must be idempotent (keyed by primary key). The PostgreSQL replication slot must be monitored because an offline consumer causes the database to retain WAL and fill the disk. Schema changes need handling, commonly Avro with Schema Registry for compatible evolution.

BASH

# Monitor PostgreSQL replication slot
SELECT slot_name, active, pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS lag_bytes
FROM pg_replication_slots
WHERE slot_name = 'debezium_accounts';

# If lag_bytes is growing and active=false: the slot is retaining WAL!
# Fix: restart the connector or drop and recreate the slot

# Key metrics:
# debezium.metrics:type=connector-metrics,name=MilliSecondsBehindSource
# debezium.metrics:type=connector-metrics,name=TotalNumberOfEventsSeen

Visual Overview

DIAGRAM

  Debezium Change Data Capture

  ┌──────────────────────────────────────────────────┐
  │  PostgreSQL Database                              │
  │                                                    │
  │  accounts table:                                   │
  │  INSERT → UPDATE → DELETE                          │
  │       │                                            │
  │       ▼                                            │
  │  Write-Ahead Log (WAL)                             │
  │  [insert SAV2001] [update SAV2001] [delete SAV2001]│
  └──────────────────────┬─────────────────────────────┘
                         │ logical replication slot
                         ▼
  ┌──────────────────────────────────────────────────┐
  │  Debezium Connector (Kafka Connect)               │
  │  Reads WAL entries in commit order                │
  │  Emits structured events:                         │
  │    op=c  before=null    after={SAV2001, 100000}   │
  │    op=u  before={100000} after={95000}             │
  │    op=d  before={95000}  after=null                │
  │    (tombstone: key=SAV2001, value=null)            │
  └──────────────────────┬─────────────────────────────┘
                         │
                         ▼
  ┌──────────────────────────────────────────────────┐
  │  Kafka Topic: cdc.public.accounts                 │
  │  Ordered change stream for downstream consumers   │
  └──────────────────────────────────────────────────┘

Key Takeaway

Debezium is an open-source CDC source connector that streams row-level changes out of a database by reading its transaction log instead of querying tables. It reads the MySQL binlog, the PostgreSQL WAL, or the equivalent for other engines, and emits every insert, update, and delete as a structured Kafka event with before and after images. Log-based capture is better than polling because it captures every change in commit order, sees deletes, adds no query load, and has millisecond latency.

At National Bank of India, the old poller had been silently missing account closures because a deleted row vanished from its query. Switching to Debezium meant every creation, balance update, and closure streamed into a Kafka topic in order within milliseconds. Downstream, a sync consumer applied them idempotently keyed by account ID, so the search index and reporting database mirrored the source exactly, including closures.

The use cases follow directly from capturing every ordered change: real-time data synchronization, the transactional outbox pattern (publishing events written in the same DB transaction as the business change), and feeding data lakes. Tombstone events after deletes drive log compaction so compacted topics correctly forget deleted keys.

Remember one sentence: Debezium turns the database's own transaction log into an ordered Kafka event stream, capturing every insert, update, and delete without polling, missed changes, or query load.

---