Why technical trace IDs alone are not enough: durable business identifiers

Property 1: Traces are short-lived and sampled.
  Most platforms retain traces for hours to 2 weeks.
  Sampling keeps only 10-20% of all traces.
  If a settlement completes on T+2 and is disputed a month later,
  the trace is long gone.

Property 2: Traces break at async boundaries.
  Service publishes to Kafka -> consumer starts a NEW trace.
  One business operation becomes many disconnected traces.
  Even with header propagation, days-long linkage is fragile.

Property 3: Trace IDs are not business-meaningful.
  You cannot query your database: "Find the trace for Rajesh's rent payment."
  The traceId was never stored as a column.

package com.bank.payment;

import java.math.BigDecimal;
import java.time.Instant;
import java.util.UUID;
import java.util.logging.Logger;

public class PaymentService {

    private static final Logger LOG = Logger.getLogger(PaymentService.class.getName());

    public String initiateTransfer(String fromId, String toId, BigDecimal amount) {
        // Generate durable business ID once at birth
        String paymentId = "PAY-" + UUID.randomUUID().toString().substring(0, 8);

        // Persist as indexed column
        // paymentRepository.save(new Payment(paymentId, fromId, toId, amount));

        // Include in every log line alongside traceId
        LOG.info("paymentId=" + paymentId + " traceId=auto Transfer initiated "
            + fromId + " -> " + toId + " amount=" + amount);

        // Include in every event published to Kafka
        // kafkaTemplate.send("payment-events", paymentId, new TransferInitiated(...));

        return paymentId;
    }
}

Expected log output:
paymentId=PAY-a1b2c3d4 traceId=xyz789 Transfer initiated SAV2001 -> SAV2002 amount=50000.00
paymentId=PAY-a1b2c3d4 traceId=xyz789 Debit completed for SAV2001
paymentId=PAY-a1b2c3d4 traceId=abc123 Settlement submitted to correspondent bank
paymentId=PAY-a1b2c3d4 traceId=def456 Settlement confirmed T+2
paymentId=PAY-a1b2c3d4 traceId=ghi789 SMS sent to Rajesh

Note: traceId changes across days, but paymentId stays the same.

package com.bank.payment.event;

import java.math.BigDecimal;
import java.time.Instant;

// Every domain event carries the business ID
public record TransferInitiated(
    String paymentId,       // durable business ID
    String fromAccountId,
    String toAccountId,
    BigDecimal amount,
    Instant initiatedAt
) {}

package com.bank.notification;

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
import java.util.logging.Logger;

@Component
public class NotificationConsumer {

    private static final Logger LOG = Logger.getLogger(NotificationConsumer.class.getName());

    @KafkaListener(topics = "payment-events", groupId = "notification-group")
    public void onTransferEvent(TransferInitiated event) {
        // Business ID from event payload - persisted, never lost
        LOG.info("paymentId=" + event.paymentId() + " Sending SMS to customer");
        // notificationRepository.save(new NotificationRecord(event.paymentId(), ...));
    }
}

Reconstruction query (months later):
SELECT * FROM payments WHERE payment_id = 'PAY-a1b2c3d4';
SELECT * FROM settlements WHERE payment_id = 'PAY-a1b2c3d4';
SELECT * FROM notifications WHERE payment_id = 'PAY-a1b2c3d4';

All traces expired, but the full story is reconstructable.

package com.bank.payment;

import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.util.logging.Logger;

@Service
public class IdempotentPaymentService {

    private static final Logger LOG = Logger.getLogger(IdempotentPaymentService.class.getName());
    private final PaymentRepository paymentRepository;

    public IdempotentPaymentService(PaymentRepository paymentRepository) {
        this.paymentRepository = paymentRepository;
    }

    public void processTransfer(String paymentId, String fromId, String toId,
                                 BigDecimal amount) {
        if (paymentRepository.existsById(paymentId)) {
            LOG.info("paymentId=" + paymentId + " Duplicate request. Returning existing result.");
            return; // Idempotent: same paymentId -> same result, no double debit
        }
        // Process the transfer
        LOG.info("paymentId=" + paymentId + " Processing new transfer");
    }
}

First request:  paymentId=PAY-a1b2c3d4 -> processed, debit of 50000.00
Retry request:  paymentId=PAY-a1b2c3d4 -> duplicate detected, no double debit

The same identifier serves three jobs:
1. Correlation (search logs)
2. Auditability (reconstruct from DB)
3. Idempotency (prevent duplicates)

package com.bank.antipattern;

// WRONG: Only log traceId. Trace expires in 48 hours.
// When customer disputes a transfer a month later, nothing to search.
public class WrongLogging {
    // LOG.info("traceId=" + traceId + " Transfer initiated");
}

// CORRECT: Log both traceId AND paymentId.
// paymentId persisted in DB, searchable forever.
public class CorrectLogging {
    // LOG.info("paymentId=" + paymentId + " traceId=" + traceId + " Transfer initiated");
}

package com.bank.payment.async;

import org.slf4j.MDC;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
import java.util.logging.Logger;

@Component
public class AsyncContextPropagation {

    private static final Logger LOG = Logger.getLogger(AsyncContextPropagation.class.getName());

    @KafkaListener(topics = "payment-events", groupId = "settlement-group")
    public void onPaymentEvent(TransferInitiated event) {
        // CORRECT: extract business ID from event and set in MDC
        MDC.put("paymentId", event.paymentId());
        try {
            LOG.info("Processing settlement for " + event.paymentId());
            // All log lines in this thread now include paymentId
        } finally {
            MDC.clear(); // clean up after processing
        }
    }
}

// WRONG: Kafka consumer logs without business ID
// traceId=abc123 Processing settlement  <-- which payment?

// CORRECT: extract paymentId from event into MDC
// paymentId=PAY-a1b2c3d4 traceId=abc123 Processing settlement