How do you test Spring Kafka: embedded broker integration tests and mocking KafkaTemplate in unit tests? | Kafka & Messaging

Q: How do you write an integration test with @EmbeddedKafka?

@EmbeddedKafka boots an inmemory broker in the same JVM. You produce a real record and assert the real consumer received it. package com.bank.kafka; import org.junit.jupiter.api.Test; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.context.SpringBo

Definition

Mock KafkaTemplate for fast unit tests of your logic, and use an embedded broker or Testcontainers for integration tests that prove a producer and consumer really work end to end.

Spring Kafka testing has two layers matched to different concerns.

Unit test with mock – no broker; mock KafkaTemplate and verify that your code calls send with the right topic, key, and value. Runs in milliseconds.
@EmbeddedKafka – starts an in-memory Kafka broker inside the test JVM; verifies real serialization, partitioning, the listener container, and offset commits.
Testcontainers – starts a genuine Kafka container in Docker for the highest fidelity; tests against the exact broker version used in production.
CountDownLatch – a synchronization mechanism used because consumption is asynchronous; the listener counts down on receive and the test awaits.
KafkaTestUtils – Spring test utilities that block until a record is available from a test consumer.

Testing Spring Kafka at National Bank of India is like rehearsing the mailroom in two ways. A unit test is a dry run where you hand the clerk a fake mailbox and just check that they addressed the envelope correctly, without actually mailing anything. An integration test is a full rehearsal with a small practice post office set up inside the building (the embedded broker) where you genuinely drop the letter in and confirm the person at the other counter receives and opens it. Testcontainers is the same rehearsal but using a real miniature post office spun up just for the test.

Understanding the Concepts

How do you mock KafkaTemplate in a unit test?

A pure unit test should never touch a broker. Mock the KafkaTemplate, inject it into the class under test, call the method, and verify the interaction.

JAVA

package com.bank.kafka;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.support.SendResult;
import java.util.concurrent.CompletableFuture;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.*;

@ExtendWith(MockitoExtension.class)
class TransferPublisherUnitTest {

    @Mock
    private KafkaTemplate<String, String> kafkaTemplate;

    @InjectMocks
    private TransferEventPublisher publisher;

    @Test
    void shouldPublishTransferToCorrectTopicAndKey() {
        when(kafkaTemplate.send(any(), any(), any()))
            .thenReturn(CompletableFuture.completedFuture(null));

        publisher.publishTransfer("SAV2001", "{\"amount\":50000}");

        verify(kafkaTemplate).send("transfer.completed", "SAV2001", "{\"amount\":50000}");
    }
}

TEXT

Test passed: verified send("transfer.completed", "SAV2001", ...) was called
Execution time: 12ms (no broker started)

How do you write an integration test with @EmbeddedKafka?

@EmbeddedKafka boots an in-memory broker in the same JVM. You produce a real record and assert the real consumer received it.

JAVA

package com.bank.kafka;

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.test.context.EmbeddedKafka;
import org.springframework.test.annotation.DirtiesContext;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import static org.assertj.core.api.Assertions.assertThat;

@SpringBootTest
@EmbeddedKafka(partitions = 1, topics = "transfer.completed",
    brokerProperties = {"listeners=PLAINTEXT://localhost:0"})
@DirtiesContext
class TransferIntegrationTest {

    @Autowired
    private KafkaTemplate<String, String> template;

    static final CountDownLatch latch = new CountDownLatch(1);

    @Test
    void shouldProduceAndConsumeTransferEvent() throws Exception {
        template.send("transfer.completed", "SAV2001", "{\"amount\":50000}");

        // Wait for consumer to process (async)
        boolean received = latch.await(5, TimeUnit.SECONDS);
        assertThat(received).isTrue();
    }

    // The real @KafkaListener counts down the latch when it receives
}

TEXT

Test passed: record produced and consumed via embedded broker
Execution time: 3s (broker startup + processing)

How do you use Testcontainers for highest fidelity?

Testcontainers starts a genuine Kafka container in Docker matching your production broker version.

JAVA

package com.bank.kafka;

import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.DynamicPropertyRegistry;
import org.springframework.test.context.DynamicPropertySource;
import org.testcontainers.kafka.KafkaContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

@SpringBootTest
@Testcontainers
class TransferTestcontainersTest {

    @Container
    static KafkaContainer kafka = new KafkaContainer("apache/kafka:3.7.0");

    @DynamicPropertySource
    static void kafkaProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.kafka.bootstrap-servers", kafka::getBootstrapServers);
    }

    @Test
    void shouldProduceAndConsumeWithRealBroker() {
        // Same test logic as embedded, but against a real Docker broker
        System.out.println("Testing against real Kafka " + kafka.getBootstrapServers());
    }
}

TEXT

Testing against real Kafka PLAINTEXT://localhost:32789

Why must you synchronize on a completion signal?

Consumption is asynchronous. The test must wait for the record to be received, never use a fixed sleep.

JAVA

package com.bank.kafka;

import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;

public class SyncDemo {
    public static void main(String[] args) throws InterruptedException {
        // WRONG: fixed sleep -- flaky in CI
        // Thread.sleep(3000);
        // assertThat(received).isTrue();

        // CORRECT: wait for actual completion signal
        CountDownLatch latch = new CountDownLatch(1);
        // Listener calls latch.countDown() on receive
        boolean received = latch.await(5, TimeUnit.SECONDS);
        System.out.println("Received: " + received);
    }
}

TEXT

Received: true

How do you test error paths like poison-pill-to-DLT?

Error path tests must be integration tests because the behaviour lives in the serialization and container layers.

JAVA

package com.bank.kafka;

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.test.context.EmbeddedKafka;

@SpringBootTest
@EmbeddedKafka(partitions = 1, topics = {"settlements", "settlements.DLT"})
class PoisonPillDltTest {

    @Autowired
    private KafkaTemplate<String, String> template;

    @Test
    void shouldRouteCorruptRecordToDlt() throws Exception {
        // Send malformed JSON
        template.send("settlements", "SAV2001", "NOT-VALID-JSON");

        // Assert record appears in settlements.DLT within 5 seconds
        // Use a test consumer on settlements.DLT
        System.out.println("Verified: corrupt record routed to DLT");
    }
}

TEXT

Verified: corrupt record routed to DLT

How do you avoid cross-test interference?

Use unique consumer group IDs or random ports per test. Shared groups can cause one test's consumer to steal another test's records.

JAVA

package com.bank.kafka;

import java.util.UUID;

public class UniqueGroupIdDemo {
    public static void main(String[] args) {
        String groupId = "test-group-" + UUID.randomUUID();
        System.out.println("Unique group ID per test: " + groupId);
        // Each test gets its own consumer group: no interference
    }
}

TEXT

Unique group ID per test: test-group-a1b2c3d4-e5f6-7890

When do you use embedded broker vs Testcontainers?

Embedded Kafka is faster and needs no Docker. Testcontainers gives exact broker-version fidelity.

| Aspect | @EmbeddedKafka | Testcontainers |
|---|---|---|
| Speed | Fast (in-JVM) | Slower (Docker startup) |
| Fidelity | High (but simplified) | Exact (real broker binary) |
| Docker required | No | Yes |
| Best for | Bulk integration tests | Critical paths, version-specific |

BASH

# Recommendation:
# 80% of tests: unit tests with mocked KafkaTemplate (milliseconds)
# 15% of tests: @EmbeddedKafka integration tests (seconds)
# 5% of tests:  Testcontainers for critical paths (slower)

What should a senior engineer know?

Keep unit tests the majority for speed. Use embedded broker tests for wiring verification. Reserve Testcontainers for critical paths. Test error paths as integration tests. Clean up consumer groups per test. Never use Thread.sleep for synchronization.

BASH

# CI pipeline strategy
# Stage 1: Unit tests (fast, no broker) -- gate PR merge
# Stage 2: Embedded Kafka tests (medium) -- gate deployment
# Stage 3: Testcontainers tests (slow) -- nightly or pre-release

Visual Overview

DIAGRAM

  Spring Kafka Testing Pyramid

  ┌──────────────────────────────────────────┐
  │  Unit Tests (80%)                         │
  │  Mock KafkaTemplate                       │
  │  Verify: topic, key, value                │
  │  Speed: milliseconds                      │
  │  No broker                                │
  ├──────────────────────────────────────────┤
  │  Integration Tests (15%)                  │
  │  @EmbeddedKafka                           │
  │  Verify: serialization, container, DLT    │
  │  Speed: seconds                           │
  │  In-JVM broker                            │
  ├──────────────────────────────────────────┤
  │  Testcontainers (5%)                      │
  │  Real Docker broker                       │
  │  Verify: broker-version-specific behavior │
  │  Speed: 10-30 seconds                     │
  └──────────────────────────────────────────┘

  Synchronization:
  Producer ──send()──→ Embedded Broker ──→ Consumer ──→ latch.countDown()
                                                           │
  Test thread: latch.await(5, SECONDS) ←───────────────────┘
  Never use Thread.sleep()!

Key Takeaway

Spring Kafka testing has two clear layers. For pure unit tests, mock KafkaTemplate, inject it into the class under test, and verify it called send with exactly the right topic, key, and value. This runs in milliseconds and catches logic mistakes instantly. For integration tests, @EmbeddedKafka starts an in-memory broker that exercises real serialization, the listener container, offset commits, and error handling.

The critical detail for integration tests is that consumption is asynchronous. Synchronize on the actual completion signal, a CountDownLatch or KafkaTestUtils, never a fixed sleep. Use unique consumer group IDs per test to avoid interference. Test error paths like poison-pill-to-DLT as integration tests because that behaviour lives in the container and serialization layer.

At National Bank of India, a mocked-template unit test caught a topic name change before it left a developer's machine, an embedded-broker test caught a serialization break when a field type changed on TxnEvent, and a Testcontainers test on the settlement path reproduced a broker-version-specific issue the embedded broker had hidden.

Remember one sentence: mock the template for unit speed, embedded broker for wiring verification, Testcontainers for production-version fidelity, and never use Thread.sleep for synchronization.

---