How do you trace a request across multiple microservices

Monolith: one stack trace
  TransferService.transfer()
    -> AccountRepository.debit()
    -> AccountRepository.credit()
    -> NotificationService.send()
  Full story in one place.

Microservices: scattered across services
  Gateway log: "transfer initiated"
  Payment Service log: "debit request received"
  Account Service log: "balance updated"
  Notification Service log: "SMS sent"
  How do you connect these? -> Correlation ID + Distributed Tracing

package com.bank.tracing;

import jakarta.servlet.Filter;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletRequest;
import jakarta.servlet.ServletResponse;
import jakarta.servlet.http.HttpServletRequest;
import org.slf4j.MDC;
import org.springframework.stereotype.Component;
import java.util.UUID;

@Component
public class CorrelationIdFilter implements Filter {

    private static final String HEADER = "X-Correlation-Id";

    @Override
    public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain)
            throws Exception {
        HttpServletRequest httpReq = (HttpServletRequest) req;
        String correlationId = httpReq.getHeader(HEADER);
        if (correlationId == null) {
            correlationId = UUID.randomUUID().toString();
        }
        MDC.put("correlationId", correlationId);
        try {
            chain.doFilter(req, res);
        } finally {
            MDC.remove("correlationId");
        }
    }
}

Expected log output (all services carry the same ID):
Gateway:      correlationId=abc-123 Transfer initiated for SAV2001
Payment Svc:  correlationId=abc-123 Debit request received
Account Svc:  correlationId=abc-123 Balance updated for SAV2001
Notification: correlationId=abc-123 SMS sent to Rajesh

package com.bank.tracing;

import io.micrometer.tracing.Tracer;
import org.springframework.stereotype.Service;
import java.math.BigDecimal;
import java.util.logging.Logger;

@Service
public class TracedTransferService {

    private static final Logger LOG = Logger.getLogger(TracedTransferService.class.getName());
    private final Tracer tracer;
    private final AccountClient accountClient;

    public TracedTransferService(Tracer tracer, AccountClient accountClient) {
        this.tracer = tracer;
        this.accountClient = accountClient;
    }

    public void transfer(String fromId, String toId, BigDecimal amount) {
        // Micrometer Tracing auto-creates spans and propagates context
        var span = tracer.nextSpan().name("transfer-saga").start();
        try (var scope = tracer.withSpan(span)) {
            LOG.info("traceId=" + span.context().traceId() + " Transfer started");
            accountClient.debit(fromId, amount);   // child span auto-created
            accountClient.credit(toId, amount);    // child span auto-created
            LOG.info("traceId=" + span.context().traceId() + " Transfer completed");
        } finally {
            span.end();
        }
    }
}

{
  "timestamp": "2024-01-15T10:30:00.000Z",
  "level": "INFO",
  "service": "payment-service",
  "traceId": "abc123",
  "spanId": "span456",
  "message": "Debit of 50000.00 from SAV2001 completed",
  "accountId": "SAV2001",
  "amount": 50000.00
}

ELK Stack flow:
Service logs (JSON) -> Filebeat/Fluentd -> Elasticsearch -> Kibana
Search: traceId:abc123 -> all log lines from all services for that request.

package com.bank.tracing;

import io.micrometer.tracing.Tracer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.header.Headers;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class TracedEventPublisher {

    private final KafkaTemplate<String, String> kafka;
    private final Tracer tracer;

    public TracedEventPublisher(KafkaTemplate<String, String> kafka, Tracer tracer) {
        this.kafka = kafka;
        this.tracer = tracer;
    }

    public void publishTransferEvent(String paymentId, String eventJson) {
        ProducerRecord<String, String> record =
            new ProducerRecord<>("payment-events", paymentId, eventJson);

        // Inject trace context into Kafka message headers
        Headers headers = record.headers();
        var span = tracer.currentSpan();
        if (span != null) {
            headers.add("traceparent",
                span.context().traceId().getBytes());
        }
        kafka.send(record);
    }
}

Without header propagation:
  Trace breaks at Kafka. Consumer starts a NEW trace.
  The two traces are unlinked.

With header propagation:
  Consumer reads traceparent from message header.
  Continues the SAME trace. Full end-to-end visibility.

# application.yml - sampling configuration
management:
  tracing:
    sampling:
      probability: 0.1  # Sample 10% of requests

Head-based sampling: decide at the start (random 10%)
  Pro: simple. Con: might miss the one slow request.

Tail-based sampling: collect all spans, keep only interesting ones
  Pro: keeps errors and slow traces. Con: more infrastructure.