You need exactly-once across a database write and a Kafka publish: why don't Kafka transactions cover this, and what is the fix (transactional outbox)? | Kafka & Messaging

Q: What does the outbox table look like?

CREATE TABLE outbox_events ( id UUID PRIMARY KEY, aggregate_id VARCHAR(50) NOT NULL, e.g., account id for ordering event_type VARCHAR(100) NOT NULL, payload TEXT NOT NULL, created_at TIMESTAMP DEFAULT NOW(), published BOOLEAN DEFAULT FALSE ); Unique constraint prevents duplicate outbox entries aggre

Definition

Kafka transactions are atomic only within Kafka, so for atomicity across a database write and a Kafka publish you use the transactional outbox -- write the business row and an outbox row in one local DB transaction, then a relay publishes the outbox at least once with an idempotency key.

Kafka transaction scope -- atomic across Kafka topic writes and consumer offset commits only; has no visibility into or control over external databases.
Dual-write problem -- writing to both a database and Kafka cannot be made atomic because they have separate transaction managers with no shared coordinator.
Transactional outbox -- in one local DB transaction, write the business row and an outbox row; a separate relay reads the outbox and publishes to Kafka.
At-least-once relay -- the relay may republish on crash, so each outbox event carries a unique id for consumer-side dedup, giving end-to-end exactly-once effect.
Relay options -- polling publisher (simple, DB-agnostic) or change data capture with Debezium (lower latency, decoupled from app code).

Imagine National Bank of India must do two things when Rajesh transfers money: update the ledger and send a notice to the clearing house. Kafka's all-or-nothing guarantee only covers the clearing house's internal paperwork. If the clerk updates the ledger and then the line to the clearing house drops, the two are out of sync. The outbox fix: in the same trip to the ledger, the clerk writes both the entry and a note in an outbox column. A separate messenger reads the outbox and sends the notice.

Understanding the Concepts

Why don't Kafka transactions cover a database write plus a Kafka publish?

A Kafka transaction gives atomicity over records written to Kafka topics and offsets committed for consumed messages. The transaction coordinator and commit markers are entirely inside Kafka. A relational database has its own completely separate transaction manager. There is no shared coordinator, and a distributed two-phase commit across both is impractical.

JAVA

package com.bank.kafka;

public class DualWriteProblem {
    public static void main(String[] args) {
        System.out.println("Naive approach 1: DB first, then Kafka");
        System.out.println("  DB commits -> crash -> Kafka never gets the event");
        System.out.println("  Result: LOST EVENT");
        System.out.println();
        System.out.println("Naive approach 2: Kafka first, then DB");
        System.out.println("  Kafka commits -> crash -> DB never gets the row");
        System.out.println("  Result: PHANTOM EVENT (consumers acted on nothing)");
        System.out.println();
        System.out.println("Both orderings are unsafe. This is the dual-write problem.");
    }
}

TEXT

Expected output:
Naive approach 1: DB first, then Kafka
  DB commits -> crash -> Kafka never gets the event
  Result: LOST EVENT

Naive approach 2: Kafka first, then DB
  Kafka commits -> crash -> DB never gets the row
  Result: PHANTOM EVENT (consumers acted on nothing)

Both orderings are unsafe. This is the dual-write problem.

How does the transactional outbox pattern work?

You give your database an outbox table. In one local transaction, write the business change and an outbox row. A relay reads unpublished rows and publishes to Kafka.

JAVA

package com.bank.service;

import java.math.BigDecimal;
import java.util.UUID;

public class TransferServiceWithOutbox {

    // @Transactional -- ONE local DB transaction
    public void transfer(String fromAccount, String toAccount, BigDecimal amount) {
        // Business rows
        // accountRepo.debit(fromAccount, amount);
        // accountRepo.credit(toAccount, amount);

        // Outbox row -- same transaction
        String eventId = UUID.randomUUID().toString();
        // outboxRepo.save(new OutboxEvent(eventId, "TransferCompleted",
        //     buildPayload(fromAccount, toAccount, amount)));

        System.out.println("DB transaction committed:");
        System.out.println("  Debit: " + fromAccount + " -" + amount);
        System.out.println("  Credit: " + toAccount + " +" + amount);
        System.out.println("  Outbox: " + eventId + " TransferCompleted");
        System.out.println("  No Kafka call here -- relay publishes from outbox");
    }

    public static void main(String[] args) {
        TransferServiceWithOutbox service = new TransferServiceWithOutbox();
        service.transfer("SAV2001", "SAV3001", new BigDecimal("50000"));
    }
}

TEXT

Expected output:
DB transaction committed:
  Debit: SAV2001 -50000
  Credit: SAV3001 +50000
  Outbox: a1b2c3d4-... TransferCompleted
  No Kafka call here -- relay publishes from outbox

What does the outbox table look like?

SQL

CREATE TABLE outbox_events (
    id          UUID PRIMARY KEY,
    aggregate_id VARCHAR(50) NOT NULL,  -- e.g., account id for ordering
    event_type  VARCHAR(100) NOT NULL,
    payload     TEXT NOT NULL,
    created_at  TIMESTAMP DEFAULT NOW(),
    published   BOOLEAN DEFAULT FALSE
);

-- Unique constraint prevents duplicate outbox entries
-- aggregate_id used as Kafka key for per-account ordering

How does the relay work?

Two common implementations: polling publisher and change data capture.

JAVA

package com.bank.relay;

public class PollingPublisher {
    // Runs on a schedule (e.g., every 100ms)
    public void publishPending() {
        // List<OutboxEvent> events = outboxRepo.findUnpublished(limit=100);
        // for (OutboxEvent event : events) {
        //     producer.send(new ProducerRecord<>("transfers",
        //         event.getAggregateId(), event.getPayload())).get();
        //     outboxRepo.markPublished(event.getId());
        // }
        System.out.println("Relay: polled 5 unpublished outbox events");
        System.out.println("  Published to Kafka, marked as published in DB");
    }

    public static void main(String[] args) {
        new PollingPublisher().publishPending();
    }
}

TEXT

Expected output:
Relay: polled 5 unpublished outbox events
  Published to Kafka, marked as published in DB

| Relay Type | How It Works | Pros | Cons |
|---|---|---|---|
| Polling publisher | Query outbox every N ms | Simple, DB-agnostic | Polling overhead, latency |
| CDC (Debezium) | Tail DB transaction log | Low latency, decoupled | More infrastructure |

Why is the relay at-least-once and how do consumers handle it?

If the relay crashes after publishing but before marking as published, it republishes on restart. Each outbox event carries a unique id that consumers use to deduplicate.

JAVA

package com.bank.kafka;

import java.util.HashSet;
import java.util.Set;

public class ConsumerDedup {
    private static final Set<String> processedEventIds = new HashSet<>();

    public static void handleEvent(String eventId, String payload) {
        if (processedEventIds.contains(eventId)) {
            System.out.println("SKIP: event " + eventId + " already processed");
            return;
        }
        processedEventIds.add(eventId);
        System.out.println("PROCESS: event " + eventId + " -> " + payload);
    }

    public static void main(String[] args) {
        handleEvent("evt-001", "TransferCompleted SAV2001->SAV3001 50000");
        handleEvent("evt-001", "TransferCompleted SAV2001->SAV3001 50000"); // Relay re-sent
    }
}

TEXT

Expected output:
PROCESS: event evt-001 -> TransferCompleted SAV2001->SAV3001 50000
SKIP: event evt-001 already processed

How does change data capture work as a relay?

Debezium tails the database transaction log and streams outbox inserts to Kafka without polling, fully decoupled from application code.

JSON

{
  "name": "outbox-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "db-primary",
    "database.dbname": "banking",
    "table.include.list": "public.outbox_events",
    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter"
  }
}

What about outbox table maintenance?

Delete or archive published rows so the table does not bloat.

SQL

-- Retention job: delete published events older than 7 days
DELETE FROM outbox_events
WHERE published = TRUE AND created_at < NOW() - INTERVAL '7 days';

What should a senior engineer know?

Order matters: the relay publishes in outbox insertion order and keys events by aggregate id for per-account ordering. The inbound side has a mirror pattern called the inbox, where a consumer records processed event ids transactionally with its state change. Kafka transactions give atomicity inside Kafka, and the outbox gives atomicity between the database and Kafka.

Visual Overview

DIAGRAM

 Transactional Outbox Pattern
 =============================

 ┌──────────────────────────────────────┐
 │           Application                │
 │                                      │
 │  @Transactional (ONE DB transaction) │
 │  ┌─────────────────────────────────┐ │
 │  │ 1. debit(SAV2001, 50000)        │ │
 │  │ 2. credit(SAV3001, 50000)       │ │
 │  │ 3. outbox.save(TransferEvent)   │ │
 │  └─────────────────────────────────┘ │
 │  COMMIT (all or nothing)             │
 └──────────────────────────────────────┘
                    │
                    ▼
 ┌──────────────────────────────────────┐
 │           Database                   │
 │                                      │
 │  accounts table: debit + credit      │
 │  outbox table:   TransferCompleted   │
 │                  published=false     │
 └──────────────┬───────────────────────┘
                │
         ┌──────┴──────┐
         ▼             ▼
 ┌──────────────┐ ┌─────────────────┐
 │ Polling      │ │ CDC (Debezium)  │
 │ Publisher    │ │ tails DB log    │
 └──────┬───────┘ └───────┬─────────┘
        │                 │
        ▼                 ▼
 ┌──────────────────────────────────────┐
 │            Kafka                     │
 │  transfers topic                     │
 │  TransferCompleted event             │
 │  (at-least-once, dedup by event id)  │
 └──────────────────────────────────────┘

Key Takeaway

Kafka transactions are atomic only within Kafka; they cannot span an external database. The naive dual write -- database then Kafka or Kafka then database -- is unsafe in both orderings because a crash in the gap leaves the two systems out of sync.

The transactional outbox pattern solves this by removing the dual write. In one local database transaction, write both the business change and an outbox row. A relay reads unpublished rows and publishes to Kafka. Because delivery is at-least-once, consumers deduplicate using the outbox event id.

In my National Bank of India transfer service, we lost events from the dual write when the broker was briefly unreachable. The outbox fixed it: the database became the single source of truth, and the relay published with dedup keys for end-to-end exactly-once effect.

Remember one sentence: Kafka transactions give atomicity inside Kafka, and the transactional outbox gives atomicity between your database and Kafka by making the database the single source of truth.

---