Observability: Logging, Tracing, Metrics

Stamp every request with a correlation ID at the edge, propagate trace and span IDs across services so a tracing tool can show the end-to-end timeline, and ship all logs to one place searchable by that ID.

Only add optional fields, never change the meaning of existing ones, deprecate before you delete, verify with contract tests, and treat a Kafka event as a stable published schema in a registry, never your internal entity.

Green services mean the parts are alive; a green business metric means the product actually works: and only the second one tells you the customer is okay.

Green infrastructure dashboards prove the building has power, not that the customer's transaction completed -- so measure business outcomes, disaggregated by tenant, region, endpoint, and workflow.

Local state breaks under a load balancer because requests scatter across instances, so externalize state to Redis or carry it in a JWT; round robin balances request count not work, so prefer least connections; and health checks must prove an instance can serve, not merely that it is alive.

The old binary cannot read data written in a new format, so you keep both formats alive through expand (add new, keep old), coexist, and only contract (remove old) once rollback is no longer needed.

Contract-first writes the spec first as the single source of truth, generates code and SDKs from it, and diffs it in CI to catch breaks early; code-first lets the contract emerge from the controllers where it can change by accident.

A trace ID answers 'what happened in this one request in the last few hours', but a business identifier answers 'what happened to this payment over its entire life': you need both.

Test failure as a contract: inject one specific failure and assert one named guarantee, rather than breaking things randomly and hoping to spot trouble.