Definition
In Amazon EKS the default VPC CNI plugin gives every Pod a real, routable IP from the VPC subnet.
- Native IP – each Pod consumes one usable subnet IP, just like an EC2 instance.
- Warm pool – the CNI pre-allocates spare IPs per node for fast Pod startup, so reserved IPs exceed active Pods.
- IP exhaustion – subnets run out of free addresses, blocking new Pods from starting.
- Fixes – add subnets or secondary CIDR, prefix delegation (/28 per ENI), tune warm pool, consider IPv6.
- Prevention – monitor subnet free-IP counts and CNI available-IP metrics proactively.
At the National Bank of India, every new teller gets their own direct phone number from the city's limited pool instead of sharing a line. That is great for reaching them directly, but the city only has so many numbers, and the phone company reserves spare numbers per department in advance. So the pool drains faster than headcount suggests, and one day a new teller cannot start because there are no numbers left.
Understanding the Concepts
Why does EKS consume IPs so quickly?
The Amazon VPC CNI assigns each Pod a real, routable VPC IP address from the node's subnet. This means every Pod consumes one usable subnet IP, just like an EC2 instance. The benefit is that Pods are directly reachable, security groups work naturally, and there is no overlay encapsulation overhead.
# Each Pod has a real VPC IP from the subnet
kubectl get pods -n prod -o wide
# NAME IP NODE
# transfer-service-x9k 10.0.1.45 ip-10-0-1-10
# transfer-service-y2m 10.0.1.46 ip-10-0-1-10
# ledger-db-0 10.0.1.47 ip-10-0-1-20
# Each IP comes from the VPC subnet 10.0.1.0/24
Why is consumption faster than Pod count suggests?
Two reasons. First, the CNI maintains a warm pool by pre-allocating spare IPs on each node ahead of demand. A node running 10 Pods may hold 20-30 reserved IPs. Multiply across many nodes and a large fraction of the subnet is reserved but idle.
Second, the number of IPs per node is bounded by instance type. Each node attaches Elastic Network Interfaces (ENIs), and each ENI holds a limited number of secondary IPs.
# Check how many IPs a node can hold
aws ec2 describe-instance-types --instance-types m5.xlarge \
--query 'InstanceTypes[0].NetworkInfo.{MaxENIs:MaximumNetworkInterfaces,IPv4PerENI:Ipv4AddressesPerInterface}'
# MaxENIs: 4, IPv4PerENI: 15
# Max IPs per node: 4 * 15 - 4 (primary IPs) = 56 Pod IPs
# But the warm pool reserves extra
kubectl get ds aws-node -n kube-system -o jsonpath='{.spec.template.spec.containers[0].env}' | jq
# WARM_IP_TARGET: 5 <- 5 spare IPs per node
# MINIMUM_IP_TARGET: 10 <- at least 10 IPs ready
What does IP exhaustion look like?
New Pods get stuck in ContainerCreating with CNI errors about failing to assign an IP, even though nodes have spare CPU and memory.
kubectl get pods -n prod
# NAME READY STATUS RESTARTS
# transfer-service-new 0/1 ContainerCreating 0
kubectl describe pod transfer-service-new -n prod
# Events:
# Warning FailedCreatePodSandBox Failed to create pod sandbox:
# rpc error: code = Unknown desc = failed to setup network:
# add cmd: failed to assign an IP address to pod
# The subnet is out of free IPs
aws ec2 describe-subnets --subnet-ids subnet-abc123 \
--query 'Subnets[0].AvailableIpAddressCount'
# 2 <- almost exhausted
How do you fix IP exhaustion?
The fixes form a toolkit applied in order of impact.
# Fix 1: Add larger or additional subnets
aws ec2 create-subnet --vpc-id vpc-123 --cidr-block 10.0.4.0/22 \
--availability-zone us-east-1a
# A /22 gives 1022 usable IPs
# Fix 2: Associate a secondary CIDR with the VPC
aws ec2 associate-vpc-cidr-block --vpc-id vpc-123 --cidr-block 100.64.0.0/16
# Then create subnets from the new CIDR for node groups
# Fix 3: Enable prefix delegation (/28 per ENI slot)
# Instead of 1 IP per slot, the CNI assigns a /28 prefix (16 IPs)
kubectl set env daemonset aws-node -n kube-system \
ENABLE_PREFIX_DELEGATION=true \
WARM_PREFIX_TARGET=1
# Dramatically increases Pods per node
# Fix 4: Tune warm pool to reduce over-allocation
kubectl set env daemonset aws-node -n kube-system \
WARM_IP_TARGET=2 \
MINIMUM_IP_TARGET=5
# Fewer spare IPs per node, slightly slower Pod startup
# Fix 5: Consider IPv6 for unlimited address space
# Create an IPv6-enabled EKS cluster
# Pods get IPv6 addresses -> exhaustion is no longer a concern
How does prefix delegation work?
Normally each ENI slot holds one secondary IP. With prefix delegation, each slot holds a /28 prefix of 16 IP addresses. This dramatically increases how many Pods a node can run.
| Mode | IPs per ENI slot | m5.xlarge (4 ENIs x 15 slots) |
|---|---|---|
| Standard | 1 IP | 56 Pod IPs |
| Prefix delegation | 16 IPs (/28) | 896 Pod IPs |
# Verify prefix delegation is active
kubectl get ds aws-node -n kube-system \
-o jsonpath='{.spec.template.spec.containers[0].env}' | jq '.[] | select(.name=="ENABLE_PREFIX_DELEGATION")'
# {"name": "ENABLE_PREFIX_DELEGATION", "value": "true"}
How do you monitor IP utilization proactively?
The right time to discover an IP shortage is during capacity planning, not when a production deploy stalls.
# Monitor subnet free IPs
aws ec2 describe-subnets --filters Name=vpc-id,Values=vpc-123 \
--query 'Subnets[*].{SubnetId:SubnetId,AZ:AvailabilityZone,FreeIPs:AvailableIpAddressCount}'
# CloudWatch metric: available IPs per subnet
# Set alarm at 80% utilization
# CNI metric: ipamd.assigned_ip_addresses vs ipamd.total_ip_addresses
# Prometheus: aws_node_ipam_available_ips < threshold
# CloudWatch alarm for subnet IP exhaustion
# Metric: available IP addresses per subnet
# Threshold: < 50 free IPs
# Action: alert the platform team
What are common mistakes?
The most common mistake is not planning IP capacity at VPC-design time. A small pilot subnet works fine with 20 Pods but fails at production scale.
# WRONG - small subnet for production
# Subnet: 10.0.1.0/24 = 251 usable IPs
# 10 nodes x 25 warm IPs = 250 IPs consumed before production load
# Result: first production scale-up fails
# CORRECT - plan large subnets from the start
# Subnet: 10.0.0.0/20 = 4091 usable IPs
# Or secondary CIDR: 100.64.0.0/16 = 65534 usable IPs
What should a senior engineer know?
This is a design consequence of the native-IP model. It must be planned at VPC-design time, not patched under pressure. The VPC CIDR, subnet sizes, instance types, and CNI settings all affect Pod density and IP consumption.
For a genuinely large banking platform, IPv6 is the strategic answer because the address space is effectively unlimited. However, IPv6 requires all internal services to support it.
# Check current IP utilization across the cluster
kubectl get nodes -o json | jq -r '
.items[] | .metadata.name + ": " +
(.status.allocatable.pods // "unknown") + " max pods"
'
Visual Overview
VPC CNI IP Assignment Model ============================= Node: m5.xlarge (4 ENIs, 15 slots each) ┌────────────────────────────────────────┐ │ ENI-1 (primary) │ │ ┌───┐ ┌───┐ ┌───┐ ... ┌───┐ ┌───┐ │ │ │IP1│ │IP2│ │IP3│ │IP14│ │IP15│ │ │ │Pod│ │Pod│ │warm│ │warm│ │warm│ │ │ └───┘ └───┘ └───┘ └───┘ └───┘ │ │ │ │ ENI-2 │ │ ┌───┐ ┌───┐ ┌───┐ ... ┌───┐ ┌───┐ │ │ │Pod│ │Pod│ │warm│ │warm│ │warm│ │ │ └───┘ └───┘ └───┘ └───┘ └───┘ │ │ │ │ Active Pods: 10 │ │ Warm/Reserved: 20 │ │ Total IPs held: 30 ← drains subnet │ └────────────────────────────────────────┘ Subnet: 10.0.1.0/24 (251 usable IPs) 10 nodes x 30 IPs = 300 IPs needed > 251 available → IP EXHAUSTION Prefix Delegation ================== Standard: 1 IP per slot → 56 Pod IPs per node Prefix: /28 (16) per slot → 896 Pod IPs per node
Key Takeaway
In Amazon EKS the VPC CNI gives every Pod a real, routable VPC IP from the subnet. That is a great design for direct reachability and native security group support, but it means each Pod consumes a usable subnet IP. The warm pool pre-allocates spare IPs on every node, so a node running 10 Pods might hold 30 reserved IPs, and across many nodes a large fraction of the subnet is consumed.
IP exhaustion is when the subnets run out, and the symptom is new Pods stuck in ContainerCreating with CNI errors even on nodes with spare CPU and memory. The fixes form a toolkit: add subnets or a secondary CIDR, enable prefix delegation to pack far more Pods per node, tune the warm pool, and consider IPv6 for unlimited addresses.
The non-negotiable practice is proactive monitoring. Watch subnet free-IP counts and the CNI available-IP metric, and alert well before utilization gets high. The worst possible moment to discover an IP shortage is when a production deploy stalls under load.
Remember one sentence: EKS gives each Pod a real VPC IP, so plan subnet sizes and enable prefix delegation at VPC-design time, and monitor free IPs proactively.
---