Kibana configuration and how the ELK stack enables centralized logging and request tracing with correlation IDs | DevOps, CI/CD & Tooling

Definition

ELK pools every service's logs into one searchable place, and a correlation ID lets you pull the complete story of a single request out of billions of log lines with one filter.

Elasticsearch – a distributed search engine that stores each log line as a JSON document in time-based indices and makes them searchable in near-real-time.
Logstash / Beats – agents that collect, parse, and ship logs from every service to Elasticsearch; Filebeat runs as a Kubernetes DaemonSet.
Kibana – a web interface for searching, visualising, and building dashboards over the logs in Elasticsearch.
Correlation ID – a unique identifier assigned to each incoming request at the gateway and propagated in headers to every downstream service, so every log line from one request shares the same ID.
MDC (Mapped Diagnostic Context) – a per-thread map in the Java logging framework that automatically attaches the correlation ID to every log line.

Imagine National Bank of India has fifty branches, each keeping its own paper logbook. When a customer complains about a failed transfer, staff would have to phone every branch and flip through separate books. ELK is like scanning every branch's logbook into one central searchable computer. The correlation ID is like stamping a single reference number on every form a customer's request touches as it moves from the teller to the back office to the vault: type that one number into Kibana and instantly see every step.

Understanding the Concepts

What is the ELK stack?

The ELK stack is three tools working together to solve the scattered-logs problem in a microservices system. Each service runs in multiple Pods, each Pod writes logs to its own ephemeral filesystem, and a single customer action may touch five services. Without centralization, diagnosing an issue means logging into many Pods by hand.

DIAGRAM

Microservices                 ELK Stack
┌──────────────┐
│ Transfer Svc │──logs──→ ┌──────────┐   ┌────────────────┐   ┌────────┐
│ Fraud Svc    │──logs──→ │ Filebeat │──→│ Elasticsearch  │──→│ Kibana │
│ Account Svc  │──logs──→ │(DaemonSet)│  │(search + index)│  │ (UI)   │
└──────────────┘          └──────────┘   └────────────────┘   └────────┘

How does Filebeat ship logs in Kubernetes?

Filebeat runs as a DaemonSet, meaning one agent per node, tailing container log files, enriching each line with metadata like Pod name and namespace, and forwarding to Elasticsearch.

YAML

# filebeat-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: logging
spec:
  selector:
    matchLabels:
      app: filebeat
  template:
    metadata:
      labels:
        app: filebeat
    spec:
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:8.12.0
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: containers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: containers
          hostPath:
            path: /var/lib/docker/containers

How does the application produce structured JSON logs?

The key enabler is that the application logs in structured JSON rather than plain text, so fields like service name, log level, timestamp, and correlation ID arrive as queryable fields.

XML

<!-- logback-spring.xml for structured JSON logging --syn-kw">>
syn-kw"><configurationsyn-kw">>
  syn-kw"><appender name="JSON" class="ch.qos.logback.core.ConsoleAppender"syn-kw">>
    syn-kw"><encoder class="net.logstash.logback.encoder.LogstashEncoder"syn-kw">>
      syn-kw"><includeMdcKeyNamesyn-kw">>correlationIdsyn-kw"></includeMdcKeyNamesyn-kw">>
      syn-kw"><customFieldssyn-kw">>{"service":"transfer-service"}syn-kw"></customFieldssyn-kw">>
    syn-kw"></encodersyn-kw">>
  syn-kw"></appendersyn-kw">>
  syn-kw"><root level="INFO"syn-kw">>
    syn-kw"><appender-ref ref="JSON" /syn-kw">>
  syn-kw"></rootsyn-kw">>
syn-kw"></configurationsyn-kw">>

TEXT

{"@timestamp":"2025-01-15T10:23:45.123Z","level":"INFO","service":"transfer-service","correlationId":"abc-123-def","message":"Transfer initiated from ACC001 to ACC002 amount 50000.00"}

What is a correlation ID and how does it flow end to end?

When a request first enters the system at the API gateway, it is assigned a unique correlation ID. That ID is placed into the MDC so every log line carries it, and it is forwarded in the headers of any outbound call to downstream services.

JAVA

package com.bank.filter;

import org.slf4j.MDC;
import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;
import java.io.IOException;
import java.util.UUID;

public class CorrelationIdFilter implements Filter {

    private static final String HEADER = "X-Correlation-Id";
    private static final String MDC_KEY = "correlationId";

    @Override
    public void doFilter(ServletRequest req, ServletResponse res,
                         FilterChain chain) throws IOException,
                         javax.servlet.ServletException {
        HttpServletRequest httpReq = (HttpServletRequest) req;
        String correlationId = httpReq.getHeader(HEADER);
        if (correlationId == null || correlationId.isBlank()) {
            correlationId = UUID.randomUUID().toString();
        }
        MDC.put(MDC_KEY, correlationId);
        try {
            chain.doFilter(req, res);
        } finally {
            MDC.remove(MDC_KEY);
        }
    }
}

JAVA

package com.bank.client;

import org.slf4j.MDC;
import org.springframework.http.HttpRequest;
import org.springframework.http.client.ClientHttpRequestExecution;
import org.springframework.http.client.ClientHttpRequestInterceptor;
import org.springframework.http.client.ClientHttpResponse;
import java.io.IOException;

public class CorrelationIdInterceptor
        implements ClientHttpRequestInterceptor {

    @Override
    public ClientHttpResponse intercept(HttpRequest request, byte[] body,
                                         ClientHttpRequestExecution exec)
            throws IOException {
        String id = MDC.get("correlationId");
        if (id != null) {
            request.getHeaders().set("X-Correlation-Id", id);
        }
        return exec.execute(request, body);
    }
}

How do you configure Kibana?

You configure Kibana by pointing it at the Elasticsearch cluster, creating an index pattern that tells Kibana which indices to read and which field is the timestamp, and then saving searches and dashboards.

YAML

# kibana.yml
server.host: "0.0.0.0"
server.port: 5601
elasticsearch.hosts: ["http://elasticsearch:9200"]
kibana.index: ".kibana"

BASH

# Create an index pattern via the Kibana API
curl -X POST "http://kibana:5601/api/saved_objects/index-pattern" \
  -H "kbn-xsrf: true" \
  -H "Content-Type: application/json" \
  -d '{
    "attributes": {
      "title": "logs-*",
      "timeFieldName": "@timestamp"
    }
  }'

How do you trace a failed request in Kibana?

When Rajesh reports a failed transfer, the engineer finds the correlation ID from the error response, types it into Kibana, and sees every log line from every service for that one request.

DIAGRAM

Kibana Discover filter:
  correlationId: "abc-123-def"

Results:
  10:23:45.123 transfer-service  INFO  Transfer initiated ACC001→ACC002 50000.00
  10:23:45.200 fraud-service     INFO  Fraud check passed for abc-123-def
  10:23:45.350 account-service   INFO  Debit ACC001 50000.00
  10:23:45.400 account-service   INFO  Credit ACC002 50000.00
  10:23:45.450 transfer-service  INFO  Transfer completed abc-123-def

How does correlation ID relate to distributed tracing?

Correlation IDs in logs give you the thread to pull, but full distributed tracing tools like Jaeger and Zipkin add timing and parent-child span relationships. A mature setup uses the W3C traceparent header so the trace ID and the log correlation ID are the same value.

JAVA

package com.bank.config;

import io.micrometer.tracing.Tracer;
import org.slf4j.MDC;
import org.springframework.stereotype.Component;

@Component
public class TraceCorrelationBridge {

    private final Tracer tracer;

    public TraceCorrelationBridge(Tracer tracer) {
        this.tracer = tracer;
    }

    public void syncTraceToMdc() {
        if (tracer.currentSpan() != null) {
            String traceId = tracer.currentSpan().context().traceId();
            MDC.put("correlationId", traceId);
        }
    }
}

What are common mistakes?

The most common mistake is logging sensitive data into Elasticsearch, where it becomes broadly searchable.

JAVA

// WRONG: logging full account number
log.info("Transfer from {} to {} amount {}",
    fullAccountNumber, targetAccount, amount);

// CORRECT: mask sensitive data
log.info("Transfer from ***{} to ***{} amount {}",
    fullAccountNumber.substring(fullAccountNumber.length() - 4),
    targetAccount.substring(targetAccount.length() - 4),
    amount);

What should a senior engineer know?

A senior engineer manages index lifecycle so logs are retained as long as compliance requires and expired to control cost. They never log passwords, full account numbers, or PII into Elasticsearch. They tie the correlation ID to the distributed trace ID so engineers can jump from a slow trace straight to the matching logs.

Visual Overview

DIAGRAM

  Request enters at API Gateway
       │
       │  assigns correlationId = UUID
       ▼
  ┌────────────────────┐  X-Correlation-Id header
  │  Transfer Service  │ ──────────────────────────→
  │  MDC: abc-123-def  │                             │
  │  log: "initiated"  │                             ▼
  └────────────────────┘                 ┌────────────────────┐
                                         │   Fraud Service    │
                                         │   MDC: abc-123-def │
                                         │   log: "check OK"  │
                                         └────────┬───────────┘
                                                   │
                                                   ▼
                                         ┌────────────────────┐
                                         │  Account Service   │
                                         │  MDC: abc-123-def  │
                                         │  log: "debit/credit"│
                                         └────────────────────┘

  All logs ship to Elasticsearch via Filebeat
       │
       ▼
  ┌──────────────────────────────────────────┐
  │              Kibana                       │
  │  Filter: correlationId = abc-123-def     │
  │  Result: complete request journey         │
  └──────────────────────────────────────────┘

Key Takeaway

The ELK stack solves the scattered-logs problem in microservices. Elasticsearch stores and indexes logs for fast search, Filebeat ships them from every Pod, and Kibana provides the search and dashboard interface. The real power comes from the correlation ID: a unique identifier assigned at the gateway and propagated to every downstream service via headers and MDC, so every log line from one request shares the same ID.

The payoff is dramatic. When Rajesh reports a failed transfer, the engineer types the correlation ID into Kibana and instantly sees every log line from every service for that one request, in time order. What used to be an hour of cross-referencing across scattered Pods becomes a ten-second filter.

The senior disciplines are: never log sensitive data like full account numbers into Elasticsearch, manage index lifecycle for compliance and cost, and tie the correlation ID to the distributed trace ID so engineers can jump from a slow trace straight to the matching logs.

Remember one sentence: ELK centralizes every service's logs into one searchable place, and the correlation ID is the single key that pulls the complete story of any request out of billions of log lines.

---