Definition
- Distributed trace – follows a single request across multiple services as a tree of spans under one trace ID.
- Client interceptor – injects the current trace context (trace ID, span ID, sampling flags) into gRPC call metadata before sending.
- Server interceptor – extracts trace context from incoming metadata and starts a child span, continuing the trace.
- OpenTelemetry – the standard instrumentation framework; handles inject and extract through propagators using W3C
traceparentformat. - Jaeger/Zipkin – backends that store and visualize the collected traces as timelines.
Imagine a transfer request at National Bank of India passing through several departments. Distributed tracing attaches one tracking number to Rajesh's request and stamps a timestamped slip at every department entry and exit. The client interceptor writes the tracking number on the envelope before sending. The server interceptor reads that number and keeps using it. Jaeger assembles all stamps into one timeline so you see exactly which department was slow.
Understanding the Concepts
What is distributed tracing?
A trace follows a single request across multiple services. It is identified by a trace ID shared across all services. Each unit of work is a span with its own span ID and a parent span ID, forming a tree. The root span starts in the first service, and each downstream call creates a child span.
Trace ID: abc-123 ├── transfer-service (root span, 150ms) │ ├── account-service (child span, 80ms) │ │ └── ledger-db (child span, 30ms) │ └── notification-service (child span, 20ms)
How does trace context travel between services?
In gRPC, the carrier is call metadata (gRPC's equivalent of HTTP headers). The client injects the active context into metadata before the call leaves, and the server extracts it from incoming metadata. The standard format is W3C traceparent, carried in a metadata key.
package com.bank.tracing;
import io.grpc.*;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Context;
import io.opentelemetry.context.propagation.TextMapSetter;
public class TracingClientInterceptor implements ClientInterceptor {
private static final TextMapSetter<Metadata> SETTER =
(carrier, key, value) -> carrier.put(
Metadata.Key.of(key, Metadata.ASCII_STRING_MARSHALLER), value);
@Override
public <Req, Resp> ClientCall<Req, Resp> interceptCall(
MethodDescriptor<Req, Resp> method,
CallOptions options,
Channel next) {
return new ForwardingClientCall.SimpleForwardingClientCall<>(
next.newCall(method, options)) {
@Override
public void start(Listener<Resp> listener, Metadata headers) {
// Inject trace context into outgoing metadata
io.opentelemetry.api.GlobalOpenTelemetry.getPropagators()
.getTextMapPropagator()
.inject(Context.current(), headers, SETTER);
super.start(listener, headers);
}
};
}
}
How does the server interceptor extract context?
The server interceptor reads trace context from incoming metadata and starts a child span. The child span becomes the current span for the duration of the handler.
package com.bank.tracing;
import io.grpc.*;
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Context;
import io.opentelemetry.context.Scope;
import io.opentelemetry.context.propagation.TextMapGetter;
public class TracingServerInterceptor implements ServerInterceptor {
private static final TextMapGetter<Metadata> GETTER =
new TextMapGetter<>() {
@Override
public Iterable<String> keys(Metadata carrier) {
return carrier.keys();
}
@Override
public String get(Metadata carrier, String key) {
var k = Metadata.Key.of(key, Metadata.ASCII_STRING_MARSHALLER);
return carrier.get(k);
}
};
private final Tracer tracer = GlobalOpenTelemetry.getTracer("bank-service");
@Override
public <Req, Resp> ServerCall.Listener<Req> interceptCall(
ServerCall<Req, Resp> call,
Metadata headers,
ServerCallHandler<Req, Resp> next) {
// Extract parent context from incoming metadata
Context parentCtx = GlobalOpenTelemetry.getPropagators()
.getTextMapPropagator()
.extract(Context.current(), headers, GETTER);
// Start a child span
Span span = tracer.spanBuilder(call.getMethodDescriptor().getFullMethodName())
.setParent(parentCtx)
.startSpan();
try (Scope scope = span.makeCurrent()) {
return next.startCall(call, headers);
} finally {
span.end();
}
}
}
How do you use OpenTelemetry's ready-made instrumentation?
In practice you rarely hand-write interceptors. OpenTelemetry ships ready-made gRPC instrumentation that registers both interceptors automatically.
<!-- Maven dependency --syn-kw">>
syn-kw"><dependencysyn-kw">>
syn-kw"><groupIdsyn-kw">>io.opentelemetry.instrumentationsyn-kw"></groupIdsyn-kw">>
syn-kw"><artifactIdsyn-kw">>opentelemetry-grpc-1.6syn-kw"></artifactIdsyn-kw">>
syn-kw"><versionsyn-kw">>2.0.0syn-kw"></versionsyn-kw">>
syn-kw"></dependencysyn-kw">>
# Run with the OpenTelemetry Java agent (auto-instruments gRPC)
java -javaagent:opentelemetry-javaagent.jar \
-Dotel.exporter.otlp.endpoint=http://collector:4317 \
-Dotel.service.name=transfer-service \
-jar transfer-service.jar
How do you configure the exporter and sampler?
The exporter sends spans to a backend. The sampler controls how many traces are recorded to balance cost and coverage.
package com.bank.config;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;
import io.opentelemetry.sdk.trace.samplers.Sampler;
public class TracingConfig {
public static OpenTelemetrySdk configure() {
OtlpGrpcSpanExporter exporter = OtlpGrpcSpanExporter.builder()
.setEndpoint("http://collector:4317")
.build();
SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
.addSpanProcessor(BatchSpanProcessor.builder(exporter).build())
.setSampler(Sampler.traceIdRatioBased(0.1)) // 10% sampling
.build();
return OpenTelemetrySdk.builder()
.setTracerProvider(tracerProvider)
.build();
}
}
How do you correlate traces with logs?
Inject the trace ID into the logging MDC so every log line links to its trace. This lets you search logs by trace ID in your logging platform.
package com.bank.service;
import io.opentelemetry.api.trace.Span;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;
public class TransferService {
private static final Logger log = LoggerFactory.getLogger(TransferService.class);
public void transfer(String from, String to, String amount) {
Span span = Span.current();
MDC.put("traceId", span.getSpanContext().getTraceId());
MDC.put("spanId", span.getSpanContext().getSpanId());
log.info("Transfer {} -> {} amount {}", from, to, amount);
// Business logic...
MDC.clear();
}
}
What are the security considerations for banking?
Never put sensitive PII into span attributes. Account numbers, balances, and customer names should not appear in traces stored in a backend like Jaeger.
// WRONG: PII in span attributes
span.setAttribute("account.number", "RAJESH-12345678");
span.setAttribute("balance", "50000.00");
// CORRECT: use safe identifiers
span.setAttribute("account.type", "savings");
span.setAttribute("transfer.amount.bucket", "10K-50K");
What about async boundaries and thread pools?
The active span context is thread-local. When work hops threads, you must capture and restore the context. OpenTelemetry provides Context.wrap() for this.
import io.opentelemetry.context.Context;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
// Wrap the task to propagate context across threads
ExecutorService pool = Executors.newFixedThreadPool(4);
Context current = Context.current();
pool.submit(current.wrap(() -> {
// Trace context is available here
processTransfer();
}));
What are common interview follow-up questions?
- What is the difference between head-based and tail-based sampling?
- How does the W3C traceparent format work?
- What happens to traces when a service does not have the tracing agent?
- How do you trace across gRPC, REST, and messaging systems in the same request?
Visual Overview
Transfer Request Flow with Tracing:
Client App
│
│ [trace_id=abc, span_id=001]
▼
┌──────────────────────┐
│ Transfer Service │ span: transfer (root)
│ Client Interceptor: │
│ inject(context, │
│ metadata) │
└──────────┬───────────┘
│ gRPC call + metadata [traceparent: abc-001]
▼
┌──────────────────────┐
│ Account Service │ span: getBalance (child of abc-001)
│ Server Interceptor: │
│ extract(metadata) │
│ startSpan(child) │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Ledger Database │ span: query (child)
└──────────────────────┘
Jaeger UI shows:
trace abc ──▶ transfer(150ms) ──▶ getBalance(80ms) ──▶ query(30ms)
Key Takeaway
Distributed tracing follows one request across services by recording a tree of spans under a single trace ID. In gRPC, a client interceptor injects the active trace context into the call's metadata, and a server interceptor extracts that context and starts a child span. OpenTelemetry handles inject and extract using the W3C traceparent format, and it ships ready-made gRPC instrumentation so you rarely write interceptors by hand.
The exporter sends spans to a backend like Jaeger or Zipkin, and the sampler controls how many traces are recorded. In production, batch the span exports and choose a sampling rate that balances cost against diagnostic coverage.
At National Bank of India, Rajesh's transfer request starts a root span in the transfer service. When the service calls the account service, the client interceptor injects the trace context into the gRPC metadata, and the account service's server interceptor extracts it and starts a child span. In Jaeger, the whole flow appears as one trace with nested spans and timings, so when a transfer is slow, Pankaj sees exactly which hop is responsible.
The senior points are to propagate context correctly across async boundaries, to never put PII into span attributes in banking, and to inject the trace ID into the logging MDC so every log line links to its trace.
Remember one sentence: a client interceptor injects trace context into gRPC metadata, a server interceptor extracts it and opens a child span, and OpenTelemetry propagates the trace end-to-end for fast diagnosis.
---