Definition
- Liveness endpoint – answers whether the process is healthy or stuck and should be killed and restarted.
- Readiness endpoint – answers whether the instance can accept traffic right now, checking real dependencies like the database.
- Heartbeat – the service periodically renews its lease with a service registry; missed heartbeats trigger eviction.
- Kubernetes probes – poll health endpoints to restart stuck containers (liveness) or remove not-ready pods from traffic (readiness).
- Gateway health check – the load balancer independently polls each backend and drops unhealthy instances from rotation.
- Monitoring and alerting – Prometheus scrapes metrics, Grafana visualizes dashboards, and PagerDuty pages the on-call engineer.
Imagine the National Bank of India wants to be sure the loans department is actually open and working. It is not enough that someone unlocked the door this morning. A supervisor walks by every few minutes and asks the clerk "are you alive and at your desk?" (liveness check) and "are you ready to take a customer right now, or still setting up?" (readiness check). The clerk also phones reception regularly to say "still here" (heartbeat to the registry). If the clerk stops answering, the supervisor sends them home and brings a replacement (Kubernetes restart), reception stops directing customers to that desk (registry eviction), and the branch manager's dashboard flashes a warning (monitoring).
Understanding the Concepts
What is the difference between liveness and readiness?
The key is distinguishing two questions. A liveness endpoint answers "is the process healthy, or is it stuck and should be killed and restarted?" A readiness endpoint answers "can this instance accept traffic right now, or is it still warming up or temporarily overloaded?" In Spring Boot, Actuator exposes /actuator/health/liveness and /actuator/health/readiness.
package com.bank.health;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;
import javax.sql.DataSource;
import java.sql.Connection;
@Component
public class DatabaseReadinessIndicator implements HealthIndicator {
private final DataSource dataSource;
public DatabaseReadinessIndicator(DataSource dataSource) {
this.dataSource = dataSource;
}
@Override
public Health health() {
try (Connection conn = dataSource.getConnection()) {
if (conn.isValid(2)) {
return Health.up()
.withDetail("database", "connected")
.build();
}
} catch (Exception e) {
// Database unreachable: not ready for traffic
}
return Health.down()
.withDetail("database", "unreachable")
.build();
}
}
# application.yml
management:
endpoint:
health:
probes:
enabled: true
group:
readiness:
include: db, diskSpace, redis
liveness:
include: livenessState
endpoints:
web:
exposure:
include: health, info, prometheus
GET /actuator/health/liveness -> {"status": "UP"} (process alive)
GET /actuator/health/readiness -> {"status": "DOWN"} (DB pool broken)
How do heartbeats work with a service registry?
After registering with a service registry like Eureka, the instance sends a periodic renew signal. If the registry misses several in a row, it marks the instance dead and evicts it. The important nuance is the staleness gap: because detection depends on missed heartbeats, there is always a small delay between an instance dying and the registry noticing.
package com.bank.config;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
import org.springframework.context.annotation.Configuration;
@Configuration
@EnableEurekaClient
public class EurekaConfig {
// Instance registers on startup and sends heartbeat every 30s
// Registry evicts after 90s of missed heartbeats
}
# application.yml - Eureka client
eureka:
instance:
lease-renewal-interval-in-seconds: 30
lease-expiration-duration-in-seconds: 90
client:
service-url:
defaultZone: http://eureka-server:8761/eureka/
Heartbeat timeline:
t=0s: instance registers with Eureka
t=30s: heartbeat sent -> registry marks instance alive
t=60s: heartbeat sent -> alive
t=90s: instance crashes, no heartbeat sent
t=120s: missed heartbeat #1
t=150s: missed heartbeat #2
t=180s: eviction -> registry removes instance from list
Staleness gap: up to 90 seconds where a dead instance is still listed
How do Kubernetes probes operationalize health endpoints?
Kubernetes probes poll the health endpoints and take automatic action. The liveness probe restarts a stuck container. The readiness probe controls whether the pod receives traffic without killing it. The startup probe holds off liveness until the app has had time to boot, preventing a slow JVM from being killed in a restart loop.
# Kubernetes deployment with all three probes
apiVersion: apps/v1
kind: Deployment
metadata:
name: account-service
spec:
template:
spec:
containers:
- name: account-service
image: bank/account-service:1.0
ports:
- containerPort: 8080
startupProbe:
httpGet:
path: /actuator/health/liveness
port: 8080
failureThreshold: 30
periodSeconds: 10
# Gives JVM up to 300s to start
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 8080
initialDelaySeconds: 0
periodSeconds: 10
failureThreshold: 3
# Restarts pod after 3 consecutive failures
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8080
initialDelaySeconds: 0
periodSeconds: 5
failureThreshold: 2
# Removes from Service endpoints after 2 failures
Probe decision matrix:
Startup probe fails -> keep trying (up to 300s), do NOT restart yet
Liveness probe fails -> restart the container (kill and recreate)
Readiness probe fails -> remove from traffic (keep container alive)
How do gateway and load-balancer health checks provide defense in depth?
The gateway or load balancer periodically checks each backend and removes any that fail from rotation. This is defense in depth: multiple layers each refuse to route to something unhealthy, so even if the registry or Kubernetes has a slight lag, the component actually routing requests independently avoids sending traffic to a bad instance.
# Spring Cloud Gateway health check for downstream services
spring:
cloud:
gateway:
discovery:
locator:
enabled: true
routes:
- id: account-service
uri: lb://account-service
predicates:
- Path=/api/accounts/**
metadata:
response-timeout: 2000
connect-timeout: 500
Defense in depth layers:
Layer 1: Eureka heartbeat -> evicts dead instance after ~90s
Layer 2: Kubernetes readiness -> removes pod from endpoints in ~10s
Layer 3: Gateway health check -> stops routing to bad backend in ~5s
Layer 4: Client-side LB -> marks instance down after 1 failure
Result: multiple independent checks catch failures at different speeds
How does monitoring close the loop?
Monitoring turns signals into human action. Metrics like up/down gauges, request error rates, latency percentiles, and probe failures are scraped by Prometheus, visualized in Grafana, and wired to alerts in PagerDuty. Synthetic checks go further by hitting the real API from outside on a schedule, catching problems that internal probes miss.
package com.bank.monitoring;
import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.stereotype.Component;
@Component
public class ServiceHealthMetrics {
public ServiceHealthMetrics(MeterRegistry registry) {
Gauge.builder("service.health.database", this,
ServiceHealthMetrics::checkDatabaseHealth)
.description("1 if database is reachable, 0 otherwise")
.register(registry);
}
private double checkDatabaseHealth() {
// Returns 1.0 for healthy, 0.0 for unhealthy
try {
// Quick DB ping
return 1.0;
} catch (Exception e) {
return 0.0;
}
}
}
Prometheus alert rules:
- alert: ServiceDown
expr: up{job="account-service"} == 0
for: 1m
labels:
severity: critical
- alert: HighErrorRate
expr: rate(http_server_requests_seconds_count{status=~"5.."}[5m]) > 0.05
for: 2m
labels:
severity: warning
- alert: ReadinessDown
expr: kube_pod_status_ready{condition="false"} == 1
for: 30s
labels:
severity: critical
What happens when readiness is down but liveness is up?
A well-built readiness check verifies real dependencies. If the database pool silently breaks, every request fails while liveness stays green. With separate probes, liveness correctly keeps the process running (restarting would not fix a broken downstream) while readiness reports not-ready and pulls the instance out of rotation.
Scenario: database pool broken
Liveness: UP (JVM process is alive and not stuck)
Readiness: DOWN (database connection pool broken)
Kubernetes action: do NOT restart, just remove from Service endpoints
Result: pod stays alive, no traffic routed to it, recovery possible
when DB pool recovers -> readiness returns UP -> traffic resumes
What are common mistakes with health checks?
The most common mistake is having a readiness check that only returns a static "UP" without verifying real dependencies. Another mistake is making liveness checks too expensive, which causes them to time out and trigger unnecessary restarts.
package com.bank.health;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;
// WRONG: liveness check that does too much
// If the DB is slow, this times out and Kubernetes restarts the pod,
// even though the process is alive. Restarting makes things worse.
@Component
public class WrongLivenessCheck implements HealthIndicator {
@Override
public Health health() {
// Do NOT check external dependencies in liveness!
// A slow DB query here causes restart loops
return Health.up().build();
}
}
package com.bank.health;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;
// CORRECT: keep liveness simple, check dependencies only in readiness
@Component
public class CorrectLivenessCheck implements HealthIndicator {
@Override
public Health health() {
// Liveness: am I stuck in a deadlock or infinite loop?
// Just check thread health, not external dependencies
if (Thread.currentThread().isInterrupted()) {
return Health.down().withDetail("reason", "interrupted").build();
}
return Health.up().build();
}
}
Wrong: liveness checks DB -> DB slow -> timeout -> pod restart -> worse
Correct: liveness checks JVM health -> readiness checks DB -> pod removed from traffic, not killed
What should a senior engineer know about health hierarchies?
Up is not a single signal but a hierarchy: the process is alive, it is ready, dependencies are healthy, end-to-end requests succeed, and someone is alerted the instant any layer degrades.
| Level | Signal | Tool | Action |
|---|---|---|---|
| Process alive | Liveness probe | Kubernetes | Restart if stuck |
| Ready for traffic | Readiness probe | Kubernetes | Remove from endpoints |
| Dependencies healthy | Custom health indicators | Actuator | Report to dashboards |
| End-to-end success | Synthetic checks | External monitor | Alert on-call |
| Trend degradation | Latency percentiles | Prometheus/Grafana | Investigate proactively |
Visual Overview
┌─────────────────────────────────────────────┐ │ Health Check Hierarchy │ │ │ │ Level 5: Synthetic E2E Check (external) │ │ │ Hits real API from outside │ │ ▼ │ │ Level 4: Monitoring & Alerting (Prometheus) │ │ │ Scrapes metrics, pages on-call │ │ ▼ │ │ Level 3: Gateway Health Check │ │ │ Drops unhealthy backends │ │ ▼ │ │ Level 2: Kubernetes Probes │ │ │ Liveness -> restart │ │ │ Readiness -> remove from traffic │ │ ▼ │ │ Level 1: Service Health Endpoints │ │ /actuator/health/liveness │ │ /actuator/health/readiness │ └─────────────────────────────────────────────┘ ┌──────────┐ heartbeat ┌──────────────┐ │ Service │ ───────────────> │ Eureka │ │ Instance │ <─ evict if miss │ Registry │ └──────────┘ └──────────────┘
Key Takeaway
Knowing a service is up means relying on objective, continuous signals rather than assuming it is fine because it started once. The most important distinction is between liveness and readiness. Liveness answers whether the process is healthy or stuck and should be restarted, while readiness answers whether the instance can actually serve traffic right now. A well-built readiness check verifies real dependencies, so readiness means genuinely able to serve, not merely running.
On top of the service's own endpoints there are heartbeats to a service registry, where the instance periodically renews its lease and the registry evicts it if several heartbeats are missed. Kubernetes probes operationalize the endpoints: the liveness probe restarts a stuck container, the readiness probe removes a not-ready pod from traffic without killing it, and a startup probe gives a slow JVM time to boot. Gateway and load-balancer health checks independently poll each backend and drop anything unhealthy from rotation, providing defense in depth.
At National Bank of India, the payment service process was alive but its database pool had silently broken, so every request failed while every basic health light stayed green. With separate liveness and readiness, liveness correctly kept the process running while readiness pulled the instance out of rotation until the dependency recovered. Monitoring with Prometheus, Grafana dashboards, and PagerDuty alerts closed the loop.
Remember one sentence: up is not a single flag but a hierarchy: alive, ready, dependencies healthy, end-to-end requests succeed, and someone is alerted the instant any layer degrades.
---