Operational documentation a production service needs: README, runbooks, troubleshooting guides, deployment docs, architecture diagrams

• README -- orients a new engineer: what the service does, key dependencies, and exact build/run/test commands.
• Runbook -- a step-by-step incident procedure tied to a specific alert, written for someone reading under stress.
• Troubleshooting guide -- a diagnostic decision tree for failures that no single alert captures.
• Deployment documentation -- how to release, verify, and roll back a deployment.
• Architecture diagram -- shows how the service connects to the gateway, database, message broker, and downstream services.
• MTTR (Mean Time To Recovery) -- the average time from an incident starting to resolution; runbooks shrink this directly.

What should a README contain?

A README answers 'what is this and how do I run it locally?' It states what the service does in one or two sentences, lists key dependencies, and gives exact commands to build, run, and test. A good README lets a new engineer go from a fresh laptop to a running service in under thirty minutes.

__CODE_BLOCK_0__
bash
mvn clean package
java -jar target/transfer-service-2.4.0.jar

__CODE_BLOCK_1__
bash
mvn clean verify

__CODE_BLOCK_2__
bash
docker compose up -d

__CODE_BLOCK_3__
bash
curl http://localhost:8080/actuator/health

What is a runbook?

A runbook is tied to a specific named alert and walks the on-call engineer through the exact steps to diagnose and fix the problem. It is written for someone who did not build the system and is reading under stress at 3 a.m.

__CODE_BLOCK_5__
bash
# Check active and pending connections
curl -s http://localhost:8080/actuator/metrics/hikaricp.connections.active \
| jq '.measurements[0].value'
curl -s http://localhost:8080/actuator/metrics/hikaricp.connections.pending \
| jq '.measurements[0].value'

# Check for long-running queries
psql -h ledger-db -U readonly -c \
"SELECT pid, now() - query_start AS duration, query
FROM pg_stat_activity
WHERE state = 'active' AND duration > interval '10 seconds'
ORDER BY duration DESC;"

## Remediation
1. If a nightly batch is holding connections, pause it:
   `kubectl scale deployment batch-processor --replicas=0`
2. If slow queries are the cause, kill the long-running query:
   `SELECT pg_terminate_backend(<pid>);`
3. If a connection leak is suspected, rolling-restart:
   `kubectl rollout restart deployment transfer-service`

## Escalation
If not resolved in 15 minutes, page the database on-call.

package com.bank.payments.health;

import com.zaxxer.hikari.HikariDataSource;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;

@Component
public class PoolHealthCheck implements HealthIndicator {

    private final HikariDataSource ds;

    public PoolHealthCheck(HikariDataSource ds) {
        this.ds = ds;
    }

    @Override
    public Health health() {
        int active = ds.getHikariPoolMXBean().getActiveConnections();
        int pending = ds.getHikariPoolMXBean().getThreadsAwaitingConnection();
        int max = ds.getMaximumPoolSize();

        if (pending > 0) {
            return Health.down()
                .withDetail("active", active)
                .withDetail("max", max)
                .withDetail("pending", pending)
                .build();
        }
        return Health.up()
            .withDetail("active", active)
            .withDetail("max", max)
            .build();
    }
}

{"status":"DOWN","details":{"active":15,"max":15,"pending":8}}

What is a troubleshooting guide?

Troubleshooting guides answer 'something is wrong but no single alert captures it.' They are more open-ended than runbooks -- a decision tree rather than a straight line.

__CODE_BLOCK_9__
bash
kubectl logs -l app=transfer-service --since=1h \
| grep "correlation-id=TXN-20240115-SAV2001"

__CODE_BLOCK_10__
bash
kafka-console-consumer --bootstrap-server kafka:9092 \
--topic account-events --from-beginning \
| grep "TXN-20240115-SAV2001"

__CODE_BLOCK_11__
bash
kubectl logs -l app=ledger-service --since=1h \
| grep "TXN-20240115-SAV2001"

## Decision Tree
- If not in transfer-service logs: transfer was never received
- If in transfer logs but not in Kafka: publishing failed
- If in Kafka but not in ledger logs: consumer lag or crash

What should deployment documentation cover?

Deployment docs cover the versioning scheme, the deploy command, how to verify success, and how to roll back. In a bank, the rollback procedure is as important as the deploy.

__CODE_BLOCK_13__
bash
# CI/CD deploys automatically on merge to main
# Manual deploy (emergency only):
kubectl set image deployment/transfer-service \
transfer-service=nbi/transfer-service:2.4.0
kubectl rollout status deployment/transfer-service

__CODE_BLOCK_14__
bash
# Health check
curl https://transfer.nbi.com/actuator/health

# Smoke test: transfer 1 rupee between test accounts
curl -X POST https://transfer.nbi.com/api/transfers \
-H 'Content-Type: application/json' \
-d '{"from":"TEST001","to":"TEST002","amount":1.00}'

__CODE_BLOCK_15__
bash
kubectl rollout undo deployment/transfer-service
# OR: deploy the previous version explicitly
kubectl set image deployment/transfer-service \
transfer-service=nbi/transfer-service:2.3.0

What should architecture diagrams show?

A single-page diagram showing how the service connects to the API gateway, database, message broker, and downstream services. This lets a new on-call engineer reason about blast radius immediately.

  Transfer Service Architecture
  ─────────────────────────────

  ┌──────────┐    ┌──────────────────┐    ┌──────────────┐
  │  API      │───▶│ Transfer Service  │───▶│ PostgreSQL    │
  │  Gateway  │    │ (payment-service) │    │ (ledger DB)   │
  └──────────┘    └────────┬─────────┘    └──────────────┘
                           │
                    publish │ consume
                           ▼
                    ┌──────────────┐
                    │  Kafka        │
                    │  (acct-events)│
                    └──────┬───────┘
                           │
               ┌───────────┼───────────┐
               ▼           ▼           ▼
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │ Fraud     │ │ Ledger   │ │ Notify   │
        │ Service   │ │ Service  │ │ Service  │
        └──────────┘ └──────────┘ └──────────┘

Why are runbooks a reliability tool?

Runbooks directly shrink MTTR (Mean Time To Recovery) because the on-call engineer does not spend the first thirty minutes rediscovering knowledge. They reduce tribal knowledge -- dangerous knowledge that lives only in one person's head. A good runbook lets any on-call engineer act confidently during an incident they have never seen.

package com.bank.ops;

/**
 * Runbooks reduce MTTR by turning tribal knowledge
 * into repeatable procedures.
 *
 * Without runbook: 30 min to diagnose + 10 min to fix = 40 min MTTR
 * With runbook:    2 min to read + 8 min to fix    = 10 min MTTR
 *
 * For NBI Transfer Service:
 * - pool-exhausted.md: 10 min avg recovery
 * - kafka-lag.md:      8 min avg recovery
 * - oom-restart.md:    5 min avg recovery
 */
public class MttrCalculator {

    public static void main(String[] args) {
        int withoutRunbook = 40;
        int withRunbook = 10;
        int improvement = withoutRunbook - withRunbook;
        System.out.println("MTTR improvement: " + improvement + " minutes");
        System.out.println("Reduction: "
            + (improvement * 100 / withoutRunbook) + "%");
    }
}

MTTR improvement: 30 minutes
Reduction: 75%

What are common mistakes?

The most common mistakes are not writing runbooks until after an outage, storing documentation in a wiki nobody updates instead of in Git, writing runbooks that are too vague to follow under stress, and not testing runbooks in game-days.

# WRONG -- vague runbook
"Check the database and fix it."

# CORRECT -- specific, copy-pasteable commands
"Run: curl localhost:8080/actuator/metrics/hikaricp.connections.active
 If active == 15 and pending > 0, the pool is exhausted.
 Step 1: pause the batch processor..."

What should a senior engineer know?

A senior engineer treats documentation as a living artifact in the repository that goes through PR review. Every incident is a chance to improve the runbook. A mature team periodically runs game-days where they deliberately trigger a failure to prove the runbook works. In a regulated bank, this discipline is how you demonstrate to auditors that the team can operate the system responsibly.