The three pillars of observability (metrics, logs, traces) and how they complement each other

• Metrics are numeric time-series that answer "how much and how fast" cheaply and continuously, such as request rate, error rate, and latency percentiles
• Logs are timestamped, discrete records of individual events that answer "what exactly happened" with full context like account IDs and stack traces
• Traces follow a single request across multiple services, answering "where did the time go and which hop failed" by producing a span waterfall
• Together they let you detect a problem (metrics), locate the failing hop (traces), and understand the root cause (logs)
• OpenTelemetry is the vendor-neutral standard that unifies all three with a shared trace ID

What are metrics?

Metrics are aggregated numbers measured over time. A metric like http_requests_total or transfer_latency_seconds is cheap to store because it is just numbers bucketed by time and a few labels, so you can keep them for a long time and query them fast. Metrics power dashboards and alerts.

# Prometheus metric exposed by the NBI transfer service
# TYPE transfer_requests_total counter
transfer_requests_total{service="transfer", status="success"} 48293
transfer_requests_total{service="transfer", status="failure"} 127

# TYPE transfer_latency_seconds histogram
transfer_latency_seconds_bucket{le="0.1"} 35000
transfer_latency_seconds_bucket{le="0.5"} 47000
transfer_latency_seconds_bucket{le="1.0"} 48100
transfer_latency_seconds_bucket{le="+Inf"} 48293

Their weakness is that they are aggregated: a metric tells you two percent of transfers failed, but not which transfers or why. You deliberately keep metric labels low-cardinality (service, status, endpoint) and you never put a customer ID or a transfer ID in a label, because that would explode storage.

What are logs?

Logs are individual event records, usually structured as JSON with a timestamp, a level, a message, and context fields. A log line can carry the full detail: account IDs, amounts, exception stack traces.

{
  "timestamp": "2024-03-15T10:05:03.412Z",
  "level": "ERROR",
  "service": "transfer-service",
  "traceId": "abc123def456",
  "message": "Transfer failed: insufficient balance",
  "fromAccount": "SAV2001",
  "toAccount": "CUR3001",
  "amount": 50000.00,
  "exception": "com.bank.core.InsufficientBalanceException"
}

Logs answer "what exactly happened to THIS request." Their weakness is volume and cost: logging every event for millions of requests is expensive, and searching raw logs across many services is slow unless they are centralized in Elasticsearch or Loki.

What are traces?

Traces solve the distributed-system problem. In a microservices bank, one transfer might touch the API gateway, the transfer service, the fraud service, the account service, and the notification service. A trace assigns one trace ID to the whole request and a span ID to each hop.

Trace ID: abc123def456

├── [gateway]          0ms ─────── 120ms  (120ms)
│   ├── [transfer-svc] 5ms ─────── 115ms  (110ms)
│   │   ├── [fraud-svc]   10ms ── 90ms    (80ms)  ◀── slow!
│   │   ├── [account-svc] 91ms ── 105ms   (14ms)
│   │   └── [notify-svc]  106ms ─ 112ms   (6ms)

Traces answer "across all these services, where did my request spend its time and which hop failed." Their weakness is that tracing every request is expensive, so teams often sample: keeping all error traces but only a fraction of successful ones.

How do the three pillars complement each other?

The standard debugging workflow chains all three through correlation:

1. A metric alert fires: error rate spiked above one percent for five minutes.
2. You open a trace for a failing request and see the fraud service span is red.
3. You pivot from the trace to the logs filtered by that exact trace ID and read the exception.

// Spring Boot: put traceId into MDC so every log line carries it
import org.slf4j.MDC;
import io.opentelemetry.api.trace.Span;

public class TransferService {
    public void processTransfer(TransferRequest request) {
        String traceId = Span.current().getSpanContext().getTraceId();
        MDC.put("traceId", traceId);
        log.info("Processing transfer from {} to {} amount {}",
                request.getFromAccount(),
                request.getToAccount(),
                request.getAmount());
        // The traceId in MDC appears in every log line automatically
    }
}

# Correlation workflow:
Metric alert ──▶ Trace waterfall ──▶ Logs by traceId
  "2% errors"     "fraud-svc slow"    "Timeout calling AML API"

How does OpenTelemetry unify the three pillars?

OpenTelemetry is the industry-standard, vendor-neutral SDK that emits metrics, logs, and traces with a shared trace ID and resource attributes, so you can jump between them seamlessly.

# OpenTelemetry Collector pipeline (otel-collector-config.yaml)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889     # Metrics to Prometheus
  loki:
    endpoint: http://loki:3100  # Logs to Loki
  jaeger:
    endpoint: jaeger:14250      # Traces to Jaeger

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]

How is metrics different from logs? (comparison table)

| Aspect | Metrics | Logs |
|---|---|---|
| Data type | Numeric time-series | Discrete text/JSON events |
| Cardinality | Low (bounded labels) | High (account IDs, trace IDs) |
| Cost | Cheap to store long-term | Expensive at high volume |
| Answers | "How much, how fast" | "What exactly happened" |
| Best for | Dashboards, alerts | Root cause investigation |
| Example | transfer_errors_total: 127 | InsufficientBalanceException for SAV2001 |

What are common mistakes with observability?

The most common mistake is having all three pillars but not correlating them. Without a shared trace ID, you cannot jump from a metric spike to the specific trace to the specific logs.

// WRONG: Logs have no trace ID -- impossible to correlate
log.error("Transfer failed for account {}", accountId);

// CORRECT: Trace ID is in MDC, appears in every log line
MDC.put("traceId", Span.current().getSpanContext().getTraceId());
log.error("Transfer failed for account {}", accountId);
// Log output: {"traceId":"abc123","message":"Transfer failed for account SAV2001"}

What should a senior engineer know?

Observability is not just "having the three pillars" but being able to ask new questions you did not anticipate. This requires high-cardinality context in logs and traces, exemplars that link a metric spike directly to a sample trace, and a culture of structured logging where every service emits the same field names.

// Exemplars: link a metric data point to a trace
import io.micrometer.core.instrument.Counter;
import io.opentelemetry.api.trace.Span;

Counter transferErrors = Counter.builder("transfer.errors")
    .tag("service", "transfer")
    .register(registry);

// When an error occurs, the exemplar links this metric point to the trace
transferErrors.increment();
// Grafana can now show: "Click this error spike to see the trace"

What are common interview follow-up questions?

• Why not just use logs for everything? Logs are expensive at scale and slow to query. Metrics are cheap and fast for dashboards and alerts. Traces solve the cross-service problem that neither metrics nor logs handle well alone.
• What is the difference between a span and a trace? A trace is the full end-to-end journey. A span is one unit of work within that journey. A trace contains many spans.
• What is tail-based sampling? A sampling strategy that keeps all traces containing errors or exceeding a latency threshold, while only sampling a fraction of fast successful traces.