Kubernetes Architecture: Control Plane, Worker Nodes, and How the Scheduler Places a Pod

• Control plane – the brain of the cluster: API server (single entry point), etcd (state store), scheduler (pod placement), controller manager (reconciliation loops).
• API server – authenticates, authorises, validates, and persists every request; the only component that talks to etcd.
• etcd – distributed, strongly consistent key-value store holding the entire desired and observed state.
• Scheduler – watches for unscheduled pods and assigns them to nodes via a filter-then-score process.
• Controller manager – runs control loops (deployment, replicaset, node controllers) that drive actual state toward desired state.
• Worker node – kubelet (pod lifecycle), kube-proxy (networking rules), container runtime (runs containers).

What is the control plane?

The control plane is the brain of the cluster. It decides what should run, where it should run, and continuously ensures reality matches the desired state.

  Control Plane
  ┌────────────────────────────────────────────┐
  │  API Server    Scheduler    Controller Mgr  │
  │       │            │              │          │
  │       └────────────┼──────────────┘          │
  │                    │                         │
  │                  etcd                        │
  │            (cluster state)                   │
  └────────────────────────────────────────────┘

What does the API server do?

The API server is the single front door for all interactions. Every request from kubectl, controllers, or kubelets goes through it. It authenticates, authorises, validates, and persists the object to etcd.

# Every kubectl command goes through the API server
kubectl get deployments -n prod
# kubectl → API server → reads from etcd → returns response

kubectl apply -f transfer-service.yaml
# kubectl → API server → validates → writes to etcd
#                       → notifies scheduler and controllers

Nothing talks to etcd directly except the API server. This keeps a single, consistent entry point and enforces access control in one place.

What is etcd?

etcd is a distributed, strongly consistent key-value store holding the entire cluster state: every deployment, pod spec, config map, and secret reference.

# etcd stores cluster state as key-value pairs
# Key: /registry/deployments/prod/transfer-service
# Value: the full Deployment spec (JSON)

# In banking, etcd is critical infrastructure:
# - Run as an odd-numbered quorum (3 or 5 members) for fault tolerance
# - Regular automated backups
# - Encryption at rest for secrets
etcdctl snapshot save /backup/etcd-$(date +%Y%m%d).db

If etcd is lost, the cluster's memory is lost. This is why it is backed up regularly and run as a high-availability quorum.

What does the scheduler do?

The scheduler watches for pods that have no node assigned and decides placement using a filter-then-score process.

# Pod requesting specific resources
apiVersion: v1
kind: Pod
metadata:
  name: transfer-service-pod
spec:
  containers:
  - name: transfer-service
    image: registry.bank.local/transfer-service:1.2.0
    resources:
      requests: { memory: "256Mi", cpu: "250m" }   # Drives scheduling
      limits:   { memory: "512Mi", cpu: "1000m" }   # Caps runtime

How does the filter-then-score process work?

Filtering (predicates): eliminate nodes that cannot run the pod.

• Not enough free CPU or memory for the pod's requests.
• Node has a taint the pod does not tolerate.
• Node does not match the pod's nodeSelector or required affinity.
• Node cannot satisfy volume or port constraints.

Scoring (priorities): rank surviving nodes and pick the best.

• How well the pod's requests fit the node's remaining resources.
• Spreading pods of the same workload across nodes and zones for resilience.
• Affinity and anti-affinity preferences.

# Example: pod anti-affinity spreads replicas across nodes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: transfer-service
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: transfer-service
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: transfer-service
              topologyKey: kubernetes.io/hostname
      containers:
      - name: transfer-service
        image: registry.bank.local/transfer-service:1.2.0
        resources:
          requests: { memory: "256Mi", cpu: "250m" }

What does the controller manager do?

The controller manager runs reconciliation loops that continuously compare desired state in etcd to observed state and take corrective action.

  Deployment Controller:
  Desired: 3 replicas of transfer-service
  Observed: 2 pods running
  Action: create 1 more pod

  Node Controller:
  Observed: node-3 not reporting for 5 minutes
  Action: mark node-3 as NotReady, reschedule its pods

# Scaling triggers the controller loop
kubectl scale deployment transfer-service --replicas=5 -n prod
# Desired: 5 replicas
# Observed: 3 pods running
# Controller creates 2 more pods, scheduler places them.

What runs on each worker node?

Kubelet: the per-node agent that watches the API server for pods assigned to its node, tells the container runtime to pull images and start containers, runs probes, and reports status.

Kube-proxy: programs networking rules (iptables or IPVS) so traffic to a Service's virtual IP is load-balanced to the right pod endpoints.

Container runtime: usually containerd, actually pulls images and runs containers via the CRI interface.

# Service: kube-proxy load-balances traffic to transfer-service pods
apiVersion: v1
kind: Service
metadata:
  name: transfer-service
spec:
  selector:
    app: transfer-service
  ports:
  - port: 80
    targetPort: 8080
# kube-proxy programs iptables rules so traffic to the Service VIP
# is distributed across all matching pods.

How do taints and tolerations control placement?

Taints repel pods from nodes. Tolerations allow specific pods to be scheduled on tainted nodes.

# Taint a node pool for payment workloads only
kubectl taint nodes payment-node-1 workload=payment:NoSchedule

# Only the payment service tolerates this taint
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  template:
    spec:
      tolerations:
      - key: workload
        operator: Equal
        value: payment
        effect: NoSchedule
      nodeSelector:
        node-pool: payment
      containers:
      - name: payment-service
        image: registry.bank.local/payment-service:1.0.0

General workloads cannot be scheduled on the payment nodes, providing isolation for sensitive workloads.

What should a senior engineer know?

• Resource requests drive scheduling; limits cap runtime. Setting requests too low overcommits nodes; too high wastes capacity.
• The control plane itself must be highly available with multiple API server replicas and an etcd quorum. A control-plane outage means you cannot deploy or self-heal.
• Topology spread constraints balance pods across availability zones for resilience.
• In banking, dedicate node pools for sensitive workloads using taints and tolerations.

# Topology spread: balance across zones
spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: transfer-service
# Ensures transfer-service pods are spread across availability zones.