Back to Kafka Streams
Q21Question 21 of 27

How does Kafka Streams handle late-arriving events with grace perio

Answer: Grace period is the extra time after a window closes during which Kafka Streams still accepts late records, window retention is how long state is kept, and late records arriving after both expire are silently dropped.

Key Concepts
  • A grace period defines how long after a window closes Kafka Streams will still accept late-arriving records for that window.
  • Window retention defines how long the windowed state store keeps data, and it must be at least window size plus grace period.
  • A record is considered late if its timestamp falls into a window that has already closed and the grace period has expired.
  • Late records that arrive after the grace period are silently dropped by default.
  • The Suppressed API holds back intermediate results and emits only the final result after the window closes and the grace period expires.
  • Stream time advances based on the maximum timestamp seen so far, not wall-clock time, so a slow partition can delay window closure.
  • Use grace periods in banking to handle delayed settlement messages and network retransmissions.

Definition

Grace period is the extra time after a window closes during which Kafka Streams still accepts late records, window retention is how long state is kept, and late records arriving after both expire are silently dropped.
  • Grace period is the duration after a window's end time during which Kafka Streams still includes late-arriving records in that window's aggregation.
  • Window retention is the total time the windowed state store keeps data; it must be at least window size + grace period.
  • Late-arriving event is a record whose timestamp falls into a window whose end time plus grace period has already passed according to stream time.
  • Stream time is the maximum record timestamp seen so far across all partitions of the input topic; it advances only when new records arrive, not by wall-clock.
  • Suppressed API buffers windowed aggregation results and emits them only after the window closes and the grace period expires, preventing intermediate updates from reaching downstream.

Think of National Bank of India's end-of-day settlement. The settlement window closes at 6:00 PM, but the bank allows a 30-minute grace period until 6:30 PM for late transaction reports from remote branches. Any transaction reported after 6:30 PM is rejected and must be processed in the next day's settlement. Window retention is how long the bank keeps the day's settlement records on file before archiving them.

Understanding the Concepts

What is a grace period in Kafka Streams?

A grace period is the extra time after a window closes during which Kafka Streams still accepts records for that window. If a 5-minute tumbling window covers [10:00, 10:05) and you set a 2-minute grace period, records with timestamps up to 10:04:59 will be accepted until stream time reaches 10:07:00.

Without a grace period, a window is final the moment stream time passes its end boundary. Any record arriving after that is dropped.

JAVA
package com.bank.streams;

import java.time.Duration;
import java.time.Instant;

public class GracePeriodExplained {
    public static void main(String[] args) {
        Duration windowSize = Duration.ofMinutes(5);
        Duration grace = Duration.ofMinutes(2);

        Instant windowStart = Instant.parse("2024-01-15T10:00:00Z");
        Instant windowEnd = windowStart.plus(windowSize);
        Instant graceDeadline = windowEnd.plus(grace);

        System.out.println("Window:  [" + windowStart + ", " + windowEnd + ")");
        System.out.println("Grace:   " + grace.toMinutes() + " minutes");
        System.out.println("Accepts late records until stream time: " + graceDeadline);
        System.out.println();

        Instant onTime = Instant.parse("2024-01-15T10:03:00Z");
        Instant lateButInGrace = Instant.parse("2024-01-15T10:04:00Z");
        Instant tooLate = Instant.parse("2024-01-15T10:04:00Z");

        System.out.println("Record at 10:03 (stream time 10:03): ACCEPTED (within window)");
        System.out.println("Record at 10:04 arriving when stream time is 10:06: ACCEPTED (within grace)");
        System.out.println("Record at 10:04 arriving when stream time is 10:08: DROPPED (past grace)");
    }
}
TEXT
Expected output:
Window:  [2024-01-15T10:00:00Z, 2024-01-15T10:05:00Z)
Grace:   2 minutes
Accepts late records until stream time: 2024-01-15T10:07:00Z

Record at 10:03 (stream time 10:03): ACCEPTED (within window)
Record at 10:04 arriving when stream time is 10:06: ACCEPTED (within grace)
Record at 10:04 arriving when stream time is 10:08: DROPPED (past grace)

What is window retention?

Window retention is how long the windowed state store keeps data after a window closes. It determines how far back you can query historical window results. Retention must be at least window size + grace period, or Kafka Streams throws a configuration error.

By default, retention is window size + 24 hours. You can override it with Materialized.withRetention().

JAVA
package com.bank.streams;

import java.time.Duration;

public class WindowRetentionExplained {
    public static void main(String[] args) {
        Duration windowSize = Duration.ofMinutes(5);
        Duration grace = Duration.ofMinutes(2);
        Duration defaultRetention = windowSize.plus(Duration.ofHours(24));
        Duration minRetention = windowSize.plus(grace);

        System.out.println("Window size:       " + windowSize.toMinutes() + " minutes");
        System.out.println("Grace period:      " + grace.toMinutes() + " minutes");
        System.out.println("Min retention:     " + minRetention.toMinutes() + " minutes");
        System.out.println("Default retention: " + defaultRetention.toHours() + " hours " + defaultRetention.toMinutesPart() + " minutes");
        System.out.println();
        System.out.println("Retention < (window + grace) causes:");
        System.out.println("  IllegalArgumentException: The retention period of the window");
        System.out.println("  store must be no smaller than its window size plus grace period.");
    }
}
TEXT
Expected output:
Window size:       5 minutes
Grace period:      2 minutes
Min retention:     7 minutes
Default retention: 24 hours 5 minutes

Retention < (window + grace) causes:
  IllegalArgumentException: The retention period of the window
  store must be no smaller than its window size plus grace period.

How does Kafka Streams decide if a record is late?

Kafka Streams uses stream time, not wall-clock time. Stream time is the maximum record timestamp seen so far across all input partitions. A record is late if its timestamp falls into a window whose end time + grace period is less than the current stream time.

This means stream time only advances when new records arrive. If no records arrive, windows stay open. A slow partition with old timestamps can delay stream time advancement.

JAVA
package com.bank.streams;

import java.time.Instant;

public class StreamTimeExplained {
    public static void main(String[] args) {
        // Simulating stream time progression
        Instant[] recordTimestamps = {
            Instant.parse("2024-01-15T10:00:00Z"),
            Instant.parse("2024-01-15T10:03:00Z"),
            Instant.parse("2024-01-15T10:06:00Z"),
            Instant.parse("2024-01-15T10:02:00Z"), // out of order
            Instant.parse("2024-01-15T10:08:00Z")
        };

        Instant streamTime = Instant.MIN;

        System.out.println("Stream time progression:");
        for (int i = 0; i < recordTimestamps.length; i++) {
            Instant ts = recordTimestamps[i];
            if (ts.isAfter(streamTime)) {
                streamTime = ts;
            }
            System.out.println("  Record " + (i + 1) + " timestamp: " + ts
                + " -> stream time: " + streamTime
                + (ts.isBefore(streamTime) ? " (LATE)" : ""));
        }
        System.out.println();
        System.out.println("Record 4 (10:02) arrived after stream time was 10:06.");
        System.out.println("Whether it is accepted depends on window + grace period.");
    }
}
TEXT
Expected output:
Stream time progression:
  Record 1 timestamp: 2024-01-15T10:00:00Z -> stream time: 2024-01-15T10:00:00Z
  Record 2 timestamp: 2024-01-15T10:03:00Z -> stream time: 2024-01-15T10:03:00Z
  Record 3 timestamp: 2024-01-15T10:06:00Z -> stream time: 2024-01-15T10:06:00Z
  Record 4 timestamp: 2024-01-15T10:02:00Z -> stream time: 2024-01-15T10:06:00Z (LATE)
  Record 5 timestamp: 2024-01-15T10:08:00Z -> stream time: 2024-01-15T10:08:00Z

Record 4 (10:02) arrived after stream time was 10:06.
Whether it is accepted depends on window + grace period.

How do you configure grace period and retention in code?

You set the grace period on the window definition and set retention on the Materialized state store.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.TimeWindows;
import org.apache.kafka.streams.kstream.Windowed;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.state.WindowStore;
import org.apache.kafka.common.serialization.Serdes;

import java.time.Duration;

public class GraceAndRetentionConfig {
    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> settlements = builder.stream("settlement-events");

        // 1-hour tumbling window with 30-minute grace period
        TimeWindows window = TimeWindows
            .ofSizeAndGrace(Duration.ofHours(1), Duration.ofMinutes(30));

        KTable<Windowed<String>, Long> counts = settlements
            .groupByKey()
            .windowedBy(window)
            .count(
                Materialized.<String, Long, WindowStore<org.apache.kafka.common.utils.Bytes, byte[]>>
                    as("settlement-counts")
                    .withRetention(Duration.ofHours(25))
                    .withKeySerde(Serdes.String())
                    .withValueSerde(Serdes.Long())
            );

        counts.toStream()
            .selectKey((wk, v) -> wk.key())
            .mapValues((k, v) -> String.valueOf(v))
            .to("settlement-counts-output");

        return builder.build();
    }

    public static void main(String[] args) {
        Topology topology = build();
        System.out.println("Settlement counting topology:");
        System.out.println("  Window:    1 hour tumbling");
        System.out.println("  Grace:     30 minutes");
        System.out.println("  Retention: 25 hours");
        System.out.println();
        System.out.println(topology.describe());
    }
}
TEXT
Expected output:
Settlement counting topology:
  Window:    1 hour tumbling
  Grace:     30 minutes
  Retention: 25 hours

Topologies:
   Sub-topology: 0
    Source: KSTREAM-SOURCE-0000000000 (topics: [settlement-events])
      --> KSTREAM-AGGREGATE-...
    Processor: KSTREAM-AGGREGATE-... (stores: [settlement-counts])
      --> KTABLE-TOSTREAM-...
    ...

What is the Suppressed API and how does it work?

By default, Kafka Streams emits an updated result every time a record is added to a window. This produces intermediate results: count=1, count=2, count=3, and so on. The Suppressed API holds back these intermediate results and emits only the final result after the window closes and the grace period expires.

This is critical in banking for end-of-day settlement totals. You want one final count per window, not a stream of intermediate updates.

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.TimeWindows;
import org.apache.kafka.streams.kstream.Windowed;
import org.apache.kafka.streams.kstream.Suppressed;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.common.serialization.Serdes;

import java.time.Duration;

public class SuppressedApiExample {
    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> settlements = builder.stream("settlement-events");

        TimeWindows window = TimeWindows
            .ofSizeAndGrace(Duration.ofHours(1), Duration.ofMinutes(30));

        KTable<Windowed<String>, Long> finalCounts = settlements
            .groupByKey()
            .windowedBy(window)
            .count(Materialized.as("settlement-counts"))
            .suppress(Suppressed.untilWindowCloses(
                Suppressed.BufferConfig.unbounded()));

        finalCounts.toStream()
            .selectKey((wk, v) -> wk.key())
            .mapValues((wk, v) -> "FINAL_COUNT=" + v)
            .to("settlement-totals");

        return builder.build();
    }

    public static void main(String[] args) {
        System.out.println("Without Suppressed:");
        System.out.println("  Emits: count=1, count=2, count=3, count=4 (intermediate)");
        System.out.println();
        System.out.println("With Suppressed.untilWindowCloses():");
        System.out.println("  Emits: count=4 (final only, after window + grace expires)");
        System.out.println();
        System.out.println("Use case: end-of-day settlement totals at National Bank of India");
    }
}
TEXT
Expected output:
Without Suppressed:
  Emits: count=1, count=2, count=3, count=4 (intermediate)

With Suppressed.untilWindowCloses():
  Emits: count=4 (final only, after window + grace expires)

Use case: end-of-day settlement totals at National Bank of India

What is a complete banking project for end-of-day settlement with late transaction handling?

This project counts settlement transactions per branch in 1-hour tumbling windows with a 10-minute grace period. It uses the Suppressed API to emit only the final count after the window closes. Late transactions arriving within the grace period are still counted.

Project structure

DIAGRAM
settlement-service/
├── pom.xml
└── src/
    ├── main/
    │   └── java/
    │       └── com/
    │           └── bank/
    │               └── streams/
    │                   ├── SettlementApp.java
    │                   ├── SettlementTopology.java
    │                   └── SettlementEvent.java
    └── test/
        └── java/
            └── com/
                └── bank/
                    └── streams/
                        └── SettlementTopologyTest.java

pom.xml

XML
<?xml version="1.0" encoding="UTF-8"?syn-kw">>
syn-kw"><project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd"syn-kw">>
    syn-kw"><modelVersionsyn-kw">>4.0.0syn-kw"></modelVersionsyn-kw">>
    syn-kw"><groupIdsyn-kw">>com.banksyn-kw"></groupIdsyn-kw">>
    syn-kw"><artifactIdsyn-kw">>settlement-servicesyn-kw"></artifactIdsyn-kw">>
    syn-kw"><versionsyn-kw">>1.0.0syn-kw"></versionsyn-kw">>
    syn-kw"><packagingsyn-kw">>jarsyn-kw"></packagingsyn-kw">>
    syn-kw"><propertiessyn-kw">>
        syn-kw"><java.versionsyn-kw">>25syn-kw"></java.versionsyn-kw">>
        syn-kw"><kafka.versionsyn-kw">>4.2.0syn-kw"></kafka.versionsyn-kw">>
        syn-kw"><maven.compiler.sourcesyn-kw">>25syn-kw"></maven.compiler.sourcesyn-kw">>
        syn-kw"><maven.compiler.targetsyn-kw">>25syn-kw"></maven.compiler.targetsyn-kw">>
    syn-kw"></propertiessyn-kw">>
    syn-kw"><dependenciessyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streamssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.apache.kafkasyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>kafka-streams-test-utilssyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>${kafka.version}syn-kw"></versionsyn-kw">>
            syn-kw"><scopesyn-kw">>testsyn-kw"></scopesyn-kw">>
        syn-kw"></dependencysyn-kw">>
        syn-kw"><dependencysyn-kw">>
            syn-kw"><groupIdsyn-kw">>org.slf4jsyn-kw"></groupIdsyn-kw">>
            syn-kw"><artifactIdsyn-kw">>slf4j-simplesyn-kw"></artifactIdsyn-kw">>
            syn-kw"><versionsyn-kw">>2.0.12syn-kw"></versionsyn-kw">>
        syn-kw"></dependencysyn-kw">>
    syn-kw"></dependenciessyn-kw">>
syn-kw"></projectsyn-kw">>

SettlementEvent.java

JAVA
package com.bank.streams;

import java.math.BigDecimal;

public class SettlementEvent {
    private String branchId;
    private String txnId;
    private BigDecimal amount;
    private String type;

    public SettlementEvent() {}

    public SettlementEvent(String branchId, String txnId,
                           BigDecimal amount, String type) {
        this.branchId = branchId;
        this.txnId = txnId;
        this.amount = amount;
        this.type = type;
    }

    public String getBranchId() { return branchId; }
    public String getTxnId() { return txnId; }
    public BigDecimal getAmount() { return amount; }
    public String getType() { return type; }

    @Override
    public String toString() {
        return branchId + "|" + txnId + "|" + amount + "|" + type;
    }

    public static SettlementEvent fromString(String value) {
        String[] parts = value.split("\\|");
        return new SettlementEvent(
            parts[0], parts[1], new BigDecimal(parts[2]), parts[3]);
    }
}

SettlementTopology.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.TimeWindows;
import org.apache.kafka.streams.kstream.Windowed;
import org.apache.kafka.streams.kstream.Suppressed;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.Topology;

import java.time.Duration;

public class SettlementTopology {

    public static Topology build() {
        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> events = builder.stream("settlement-events");

        // 1-hour window with 10-minute grace for late transactions
        TimeWindows window = TimeWindows
            .ofSizeAndGrace(Duration.ofHours(1), Duration.ofMinutes(10));

        KTable<Windowed<String>, Long> finalCounts = events
            .groupByKey()
            .windowedBy(window)
            .count(Materialized.as("settlement-counts"))
            .suppress(Suppressed.untilWindowCloses(
                Suppressed.BufferConfig.unbounded()));

        finalCounts.toStream()
            .selectKey((wk, v) -> wk.key())
            .mapValues((wk, v) -> "SETTLEMENT_TOTAL|count=" + v)
            .to("settlement-totals");

        return builder.build();
    }
}

SettlementApp.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;

import java.util.Properties;
import java.util.concurrent.CountDownLatch;

public class SettlementApp {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "settlement-service");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        KafkaStreams streams = new KafkaStreams(
            SettlementTopology.build(), props);
        CountDownLatch latch = new CountDownLatch(1);

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            streams.close();
            latch.countDown();
        }));

        try {
            streams.start();
            System.out.println("Settlement service started");
            System.out.println("  Input:  settlement-events");
            System.out.println("  Output: settlement-totals");
            System.out.println("  Window: 1 hour, grace: 10 minutes");
            System.out.println("  Mode:   final count only (Suppressed)");
            latch.await();
        } catch (Exception e) {
            System.exit(1);
        }
    }
}

SettlementTopologyTest.java

JAVA
package com.bank.streams;

import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.time.Instant;
import java.util.Properties;

public class SettlementTopologyTest {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                  Serdes.String().getClass());

        try (TopologyTestDriver driver =
                 new TopologyTestDriver(SettlementTopology.build(), props)) {

            TestInputTopic<String, String> input = driver.createInputTopic(
                "settlement-events", new StringSerializer(), new StringSerializer());
            TestOutputTopic<String, String> output = driver.createOutputTopic(
                "settlement-totals",
                new StringDeserializer(), new StringDeserializer());

            Instant hour1Start = Instant.parse("2024-01-15T10:00:00Z");

            // 3 on-time events in hour 1
            input.pipeInput("BRANCH_MUMBAI", "BRANCH_MUMBAI|TXN001|50000|NEFT", hour1Start);
            input.pipeInput("BRANCH_MUMBAI", "BRANCH_MUMBAI|TXN002|75000|RTGS", hour1Start.plusSeconds(1800));
            input.pipeInput("BRANCH_MUMBAI", "BRANCH_MUMBAI|TXN003|30000|IMPS", hour1Start.plusSeconds(3000));

            // Late event within grace (arrives 5 min after window closes)
            input.pipeInput("BRANCH_MUMBAI", "BRANCH_MUMBAI|TXN004|20000|NEFT",
                hour1Start.plusSeconds(2500));

            // Advance stream time past window + grace to trigger Suppressed emit
            Instant triggerTime = hour1Start.plusSeconds(4200); // 1h10m
            input.pipeInput("BRANCH_MUMBAI", "BRANCH_MUMBAI|TXN005|10000|NEFT", triggerTime);

            System.out.println("Settlement totals (final only):");
            while (!output.isEmpty()) {
                System.out.println("  " + output.readValue());
            }
            System.out.println();
            System.out.println("4 events counted in hour 1 (including late TXN004)");
            System.out.println("Emitted only after window + grace expired");
        }
    }
}
TEXT
Expected output:
Settlement totals (final only):
  SETTLEMENT_TOTAL|count=4

4 events counted in hour 1 (including late TXN004)
Emitted only after window + grace expired

How to run the project

BASH
# Step 1: Start Kafka
docker-compose up -d kafka zookeeper

# Step 2: Create topics
kafka-topics.sh --create --topic settlement-events --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic settlement-totals --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

# Step 3: Build
cd settlement-service
mvn clean package

# Step 4: Run
java -cp target/settlement-service-1.0.0.jar com.bank.streams.SettlementApp

# Step 5: Produce settlement events
kafka-console-producer.sh --topic settlement-events --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
# BRANCH_MUMBAI:BRANCH_MUMBAI|TXN001|50000|NEFT
# BRANCH_MUMBAI:BRANCH_MUMBAI|TXN002|75000|RTGS

# Step 6: Consume totals
kafka-console-consumer.sh --topic settlement-totals --from-beginning --bootstrap-server localhost:9092

What are common mistakes with grace periods and late events?

The most common mistake is setting no grace period and wondering why late records are dropped.

Wrong: no grace period, late records silently dropped

JAVA
package com.bank.streams;

public class WrongNoGracePeriod {
    public static void main(String[] args) {
        System.out.println("WRONG: Using ofSizeWithNoGrace()");
        System.out.println("  TimeWindows.ofSizeWithNoGrace(Duration.ofHours(1))");
        System.out.println();
        System.out.println("What happens:");
        System.out.println("  Window [10:00, 11:00) closes at stream time 11:00");
        System.out.println("  Late transaction at 10:55 arriving at stream time 11:01");
        System.out.println("  Result: SILENTLY DROPPED, no error, no log");
        System.out.println();
        System.out.println("In banking, this means lost settlements.");
        System.out.println("The branch's late transaction is never counted.");
    }
}
TEXT
Expected output:
WRONG: Using ofSizeWithNoGrace()
  TimeWindows.ofSizeWithNoGrace(Duration.ofHours(1))

What happens:
  Window [10:00, 11:00) closes at stream time 11:00
  Late transaction at 10:55 arriving at stream time 11:01
  Result: SILENTLY DROPPED, no error, no log

In banking, this means lost settlements.
The branch's late transaction is never counted.

Correct: set grace period matching your SLA

JAVA
package com.bank.streams;

public class CorrectGracePeriod {
    public static void main(String[] args) {
        System.out.println("CORRECT: Using ofSizeAndGrace()");
        System.out.println("  TimeWindows.ofSizeAndGrace(Duration.ofHours(1), Duration.ofMinutes(30))");
        System.out.println();
        System.out.println("What happens:");
        System.out.println("  Window [10:00, 11:00) accepts late records until stream time 11:30");
        System.out.println("  Late transaction at 10:55 arriving at stream time 11:10");
        System.out.println("  Result: ACCEPTED, counted in the window");
        System.out.println();
        System.out.println("Set grace period to match your SLA for late arrivals.");
        System.out.println("In banking: 30 minutes covers remote branch network delays.");
    }
}
TEXT
Expected output:
CORRECT: Using ofSizeAndGrace()
  TimeWindows.ofSizeAndGrace(Duration.ofHours(1), Duration.ofMinutes(30))

What happens:
  Window [10:00, 11:00) accepts late records until stream time 11:30
  Late transaction at 10:55 arriving at stream time 11:10
  Result: ACCEPTED, counted in the window

Set grace period to match your SLA for late arrivals.
In banking: 30 minutes covers remote branch network delays.

What should a senior or principal engineer know?

At the senior level, you must understand the trade-offs of grace periods and suppression.

  • Grace period = latency: a longer grace period means results are delayed longer before being emitted (when using Suppressed). For a 30-minute grace, the final result arrives 30 minutes after the window closes.
  • Suppressed buffer memory: Suppressed.untilWindowCloses() buffers all intermediate results in memory. Use BufferConfig.maxBytes(1_000_000) to limit memory usage. When the buffer is full, the oldest window is forced out early.
  • Stream time vs wall-clock time: if all producers stop, stream time freezes and no windows close. Use a heartbeat producer that sends periodic records with current wall-clock timestamps to advance stream time.
  • Dropped records metric: monitor kafka.streams:type=stream-task-metrics,name=dropped-records-total to detect late records being dropped.
  • Multi-partition stream time: stream time is tracked per task (per partition). A fast partition may have stream time far ahead of a slow partition, causing records on the slow partition to be considered late even if they are on time by wall-clock.
JAVA
package com.bank.streams;

public class SeniorGracePeriodKnowledge {
    public static void main(String[] args) {
        System.out.println("Senior/Principal grace period checklist:");
        System.out.println("  1. Grace period adds latency to Suppressed output");
        System.out.println("  2. Suppressed buffers consume heap memory");
        System.out.println("  3. Use BufferConfig.maxBytes() to cap memory");
        System.out.println("  4. Stream time freezes when no records arrive");
        System.out.println("  5. Use heartbeat producer to advance stream time");
        System.out.println("  6. Monitor dropped-records-total metric via JMX");
        System.out.println("  7. Multi-partition: slow partitions cause false 'late'");
        System.out.println("  8. Retention must be >= window size + grace period");
    }
}
TEXT
Expected output:
Senior/Principal grace period checklist:
  1. Grace period adds latency to Suppressed output
  2. Suppressed buffers consume heap memory
  3. Use BufferConfig.maxBytes() to cap memory
  4. Stream time freezes when no records arrive
  5. Use heartbeat producer to advance stream time
  6. Monitor dropped-records-total metric via JMX
  7. Multi-partition: slow partitions cause false 'late'
  8. Retention must be >= window size + grace period

What are common interview follow-up questions?

  • What happens to a record that is dropped as late? (It is silently discarded. No exception is thrown. The dropped-records-total metric increments.)
  • Can you process late records in a separate topic? (Not built in. You would need a custom Transformer that checks timestamps and forwards late records to a side output topic.)
  • What is the difference between grace period and retention? (Grace period controls how long late records are accepted. Retention controls how long state is kept for queries. Retention must be >= window size + grace.)
  • Does Suppressed work with session windows? (Yes. Suppressed.untilWindowCloses() works with tumbling, hopping, and session windows.)
  • How do you test late records with TopologyTestDriver? (Pipe an on-time record, then pipe a record with a timestamp that advances stream time past the window + grace, then pipe a late record and verify it does not appear in the output.)

Visual Overview

DIAGRAM
 Late Event Handling Timeline
 =============================

 Window: [10:00, 11:00)  Grace: 30 min

  10:00         11:00        11:30
    │──── Window ────│── Grace ──│
    │                │           │
    │  On-time       │  Late but │  Too late
    │  records       │  accepted │  DROPPED
    │                │           │
    ▼                ▼           ▼
 ┌─────────────┐ ┌────────┐ ┌──────────┐
 │ Counted in  │ │ Still  │ │ Silently │
 │ window      │ │ counted│ │ dropped  │
 │ aggregation │ │ in     │ │ not in   │
 │             │ │ window │ │ any      │
 └─────────────┘ └────────┘ └──────────┘


 Suppressed API
 ===============

 Without Suppressed:         With Suppressed:
 ┌─────────────────┐        ┌─────────────────┐
 │ count=1 (emit)  │        │ count=1 (buffer) │
 │ count=2 (emit)  │        │ count=2 (buffer) │
 │ count=3 (emit)  │        │ count=3 (buffer) │
 │ count=4 (emit)  │        │ count=4 (buffer) │
 └─────────────────┘        │  ...wait grace...│
  4 downstream records      │ count=4 (EMIT)  │
                            └─────────────────┘
                             1 downstream record

Key Takeaway

Grace period, window retention, and the Suppressed API work together to handle late-arriving events in Kafka Streams. A grace period defines how long after a window closes the system still accepts records for that window. Window retention defines how long state is kept in the store. Late records that arrive after the grace period expires are silently dropped.

In our settlement service at National Bank of India, we used 1-hour tumbling windows with a 10-minute grace period. Settlement transactions from remote branches sometimes arrived a few minutes late due to network delays. The grace period ensured these late transactions were still counted. The Suppressed API held back intermediate counts and emitted only the final settlement total after the window and grace period expired.

The most common mistake is using ofSizeWithNoGrace() in production. This means any record arriving even one millisecond after the window closes is dropped. In banking, this leads to incorrect settlement totals. Always set a grace period that matches your SLA for late arrivals.

At the senior level, understand that grace period adds latency to Suppressed output, Suppressed buffers consume heap memory, and stream time only advances when records arrive. If producers stop, windows stay open forever. Use a heartbeat producer to advance stream time and monitor the dropped-records-total metric to detect lost records.

Remember one sentence: grace period keeps the window open for latecomers, Suppressed holds back intermediate results, and together they ensure your final windowed count includes every record that arrives within your SLA.

---

Share
Use ← → arrow keys to navigate