Definition
- Trace – represents one request's whole journey, identified by a unique traceId, composed of a tree of timed spans.
- Span – one unit of work within a trace, with its own spanId, a parent, and start/end timestamps.
- Context propagation – the gateway injects trace context into request headers (W3C
traceparent), and downstream services extract and continue the same trace. - Micrometer Tracing – the Spring successor to Sleuth; automatically generates, propagates, and logs trace IDs.
- Sampling – tracing every request is expensive, so you sample a fraction; tail-based sampling keeps errored/slow traces.
Imagine Rajesh's loan application at National Bank of India gets a single reference number stamped at the front desk. As the application moves from verification to credit scoring to approval, every department writes its notes under that same reference number and records when it received and finished the file. Later, a manager pulls that one reference number and sees the complete journey: which department had it, for how long, and where it got stuck. The trace ID is that reference number, and each department's timed note is a span.
Understanding the Concepts
What is a trace and a span?
A trace represents one request's whole journey. Within a trace, each unit of work is a span with its own span ID, a parent span ID, and start/end timestamps. A trace is a tree of spans sharing one trace ID.
// Trace: traceId = abc-123
// ├── Span 1: Gateway (root span, 0-120ms)
// │ ├── Span 2: Payment Service (5-80ms)
// │ │ └── Span 3: Account Service (10-50ms)
// │ └── Span 4: Notification Service (85-115ms)
// Each span records:
// - spanId (unique)
// - parentSpanId (links to caller)
// - operationName ("POST /api/v1/transfers")
// - startTime, endTime
// - tags (accountId=SAV2001, status=200)
How does context propagation work?
The gateway injects the trace context into request headers using the W3C Trace Context standard. Downstream services extract it and continue the same trace.
// W3C traceparent header format:
// traceparent: 00-<traceId>-<parentSpanId>-<flags>
// Example:
// traceparent: 00-abc123def456-span001-01
// Gateway sends request to Payment Service:
// Headers:
// traceparent: 00-abc123def456-gateway-span-01
// Payment Service receives, extracts traceId, creates child span:
// traceId = abc123def456 (same!)
// parentSpanId = gateway-span
// spanId = payment-span (new)
// Payment Service calls Account Service:
// Headers:
// traceparent: 00-abc123def456-payment-span-01
// All services share the same traceId
How does Micrometer Tracing work in Spring?
Micrometer Tracing, the successor to Spring Cloud Sleuth, handles trace propagation automatically. It puts the trace ID into the logging context so every log line carries it.
<!-- pom.xml dependencies --syn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>io.micrometersyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>micrometer-tracing-bridge-bravesyn-kw"></artifactIdsyn-kw">>
syn-kw"></dependencysyn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>io.zipkin.reporter2syn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>zipkin-reporter-bravesyn-kw"></artifactIdsyn-kw">>
syn-kw"></dependencysyn-kw">>
# application.yml
management:
tracing:
sampling:
probability: 0.1 # Sample 10% of requests
zipkin:
tracing:
endpoint: http://zipkin:9411/api/v2/spans
// With Micrometer Tracing, log output automatically includes traceId: // Gateway: [traceId=abc123] Routing POST /api/v1/transfers // Payment Svc: [traceId=abc123] Processing transfer TXN-456 // Account Svc: [traceId=abc123] Debiting SAV2001 by 50000 // Notification: [traceId=abc123] Sending SMS to Priya // Search logs for traceId=abc123 → every log from every service
How does Zipkin visualize traces?
Zipkin receives spans from all services and draws a waterfall timeline showing latency per service.
// Zipkin waterfall for traceId=abc123: // |-- Gateway (0-120ms) ──────────────────────────────────────| // |-- Payment Service (5-80ms) ─────────────────────| | // |-- Account Service (10-50ms) ──────| | | // |-- Notification Service (85-115ms) ──────────| | | // Immediately visible: Account Service took 40ms of the 80ms payment call // Without tracing: "the transfer is slow" → which service? Unknown.
What is OpenTelemetry and why use it?
OpenTelemetry is the vendor-neutral standard for traces, metrics, and logs. Using it avoids lock-in to one tracing vendor.
// With OpenTelemetry:
// Services use OpenTelemetry SDK → produce spans
// Spans sent to OpenTelemetry Collector (centralized)
// Collector exports to Zipkin, Jaeger, Datadog, or any backend
// Advantage: switch from Zipkin to Jaeger → change collector config only
// No code changes in any service
How does sampling control cost?
Tracing every request in a high-throughput bank produces overwhelming volume. Sampling traces only a fraction.
// Head-based sampling: decide at the gateway
// probability: 0.1 → trace 10% of requests
// Decision encoded in traceparent flags → consistent across all services
// If gateway says "sample", all services in that trace produce spans
// If gateway says "don't sample", no service produces spans
// Tail-based sampling: decide at the collector
// Collect ALL spans temporarily
// Keep only traces that: errored, exceeded latency threshold, or matched rules
// Advantage: always captures the interesting traces (errors, slow requests)
// Cost: collector needs more memory to buffer all spans temporarily
What is the reactive tracing subtlety?
In a reactive gateway, the request hops across threads. The trace context cannot live in a plain thread-local; it must be carried through the reactive context.
// In traditional Spring MVC:
// Thread-local holds the trace context → works because one thread per request
// In Spring WebFlux (reactive):
// Request hops across threads in the event loop
// Thread-local trace context gets lost on the hop!
// Fix: Micrometer Tracing + Reactor Context integration
// Trace context rides the Reactor subscriber context
// Survives thread switches automatically
// Common bug: using a non-context-aware library that breaks propagation
What are common mistakes? (wrong vs correct)
// WRONG: no trace propagation between services
// Gateway logs: traceId=abc123
// Payment logs: traceId=xyz789 (new trace! not connected!)
// Cannot follow the request across services
// CORRECT: propagate via W3C traceparent headers
// Gateway logs: traceId=abc123
// Payment logs: traceId=abc123 (same trace, child span)
// All logs connected under one traceId
// WRONG: sampling 100% in production
// 50,000 requests/second × 4 spans each = 200,000 spans/second
// Zipkin overwhelmed, storage costs explode
// CORRECT: sample 10% for routine, tail-sample for errors
// 5,000 traces/second (routine) + all error traces
What should a senior engineer know?
A senior engineer propagates baggage (business context like accountId) carefully for debugging, implements tail-based sampling to always capture error traces, and uses the cheapest first step (log the traceId) even before a full tracing backend is set up.
// Senior tracing checklist:
// 1. Propagate traceId via W3C traceparent (standard, cross-language)
// 2. Log traceId in every service (MDC integration with Micrometer)
// 3. Sample wisely: head-based for routine, tail-based for errors
// 4. Use OpenTelemetry to avoid vendor lock-in
// 5. Monitor span export latency and drop rates
// 6. Propagate baggage (accountId) carefully, never PII
Visual Overview
Distributed Tracing Flow
────────────────────────
Client Gateway Payment Svc Account Svc
│ │ │ │
│── POST /transfer ───▶│ │ │
│ │ mint traceId=abc123 │ │
│ │ create root span │ │
│ │ │ │
│ │── traceparent:abc123 ─▶│ │
│ │ │ create child span │
│ │ │── traceparent:abc123 ──▶│
│ │ │ │ create child span
│ │ │◀── response ───────│
│ │◀── response ──────────│ │
│◀── response ─────────│ │ │
│ │ │ │
│ All spans exported to Zipkin with traceId=abc123 │
Zipkin Waterfall
────────────────
|── Gateway ──────────────────────────────────────|
|── Payment Service ─────────────────| |
|── Account Service ───────| | |
0ms 10ms 50ms 80ms 120ms
Key Takeaway
Distributed tracing follows one request as it travels through many services, and the gateway is where the trace begins. A trace has a single traceId, and within it every unit of work is a span with its own spanId, parent, and timing. The gateway mints the traceId and creates the root span, then propagates the trace context to every downstream service via the W3C traceparent header. All spans are exported to a backend like Zipkin, which draws the end-to-end waterfall.
At National Bank of India, this is how Pankaj diagnosed a customer complaint that a transfer took eight seconds. The request passed through the gateway, the payment service, the account service, and the notification service. Before tracing, each service's logs were separate islands. With tracing, they searched for one traceId and immediately saw the waterfall: the account service debit call was the bottleneck. Without tracing they would have been guessing.
Sampling controls cost because tracing every request at 50,000 requests per second would overwhelm the tracing backend. Head-based sampling decides at the gateway and is encoded in the traceparent flags so the whole trace is consistently sampled. Tail-based sampling in the collector keeps all errored or slow traces, which are the ones you actually need. Even without a full tracing backend, simply logging a traceId in every service is the cheapest useful first step.
Remember one sentence: the gateway mints a traceId, propagates it via W3C traceparent headers, every service adds timed spans, and Zipkin stitches them into one end-to-end timeline you can search and visualize.
---