Definition
- Subnet CIDR immutability – a subnet's address range is fixed at creation and cannot be resized because it is already allocated and in use by network interfaces.
- Secondary CIDR – an additional CIDR block associated with the existing VPC, from which new, larger subnets can be created.
- Gradual migration – workloads move to new subnets without downtime by adding new subnets to ASGs, EKS clusters, or DB subnet groups and draining the old ones.
- 100.64.0.0/10 range – a common secondary CIDR for EKS pod IPs using custom networking, keeping pods off the scarce routable space.
A subnet is like a floor at National Bank of India with a fixed set of room numbers, say 101 to 150, and staff are already sitting in those rooms. You cannot declare the floor now runs 101 to 300 because the walls are built and people are working. Instead you lease a whole new floor (a secondary CIDR), build new rooms there, and move teams over one at a time while the old floor keeps working.
Understanding the Concepts
Why can't you resize a subnet CIDR in place?
A subnet CIDR is immutable. When you create a /24 subnet you get 251 usable addresses, and AWS allocates that range to the subnet. Resources attach elastic network interfaces with IPs from it. You cannot expand it to a /23 because the adjacent address space may already belong to another subnet, and resizing would require renumbering interfaces that are actively serving traffic.
// Illustrating why resize is impossible
package com.bank.infrastructure;
public class SubnetCidrImmutability {
public static void main(String[] args) {
System.out.println("Subnet: 10.0.1.0/24 (251 usable IPs)");
System.out.println("Adjacent: 10.0.2.0/24 (belongs to another subnet)");
System.out.println();
System.out.println("Cannot expand to 10.0.1.0/23 because:");
System.out.println(" 1. 10.0.2.0/24 is already allocated");
System.out.println(" 2. Live ENIs have IPs from the current range");
System.out.println(" 3. Renumbering would disrupt all running services");
}
}
How do you associate a secondary CIDR?
You add capacity at the VPC level. The VPC can hold several CIDR blocks. You associate a secondary CIDR block, then create new, larger subnets from it in each Availability Zone.
# Associate a secondary CIDR with the VPC
aws ec2 associate-vpc-cidr-block --vpc-id vpc-0abc --cidr-block 100.64.0.0/16
# Create new subnets from the secondary CIDR
aws ec2 create-subnet --vpc-id vpc-0abc --cidr-block 100.64.0.0/20 --availability-zone ap-south-1a
aws ec2 create-subnet --vpc-id vpc-0abc --cidr-block 100.64.16.0/20 --availability-zone ap-south-1b
// Conceptual: adding a secondary CIDR and creating subnets
package com.bank.infrastructure;
public class SecondaryCidrExpansion {
public static void main(String[] args) {
System.out.println("VPC original: 10.0.0.0/16 (app subnets nearly full)");
System.out.println("Associate secondary: 100.64.0.0/16");
System.out.println("Create subnets: 100.64.0.0/20 (AZ-a), 100.64.16.0/20 (AZ-b)");
System.out.println("Each /20 gives 4091 usable IPs -- plenty of room.");
}
}
How do you migrate workloads gradually?
Migration is gradual and non-disruptive. You never renumber a running interface; you drain the old subnet and fill the new one.
// Migration strategies per workload type
package com.bank.infrastructure;
public class WorkloadMigration {
public static void main(String[] args) {
System.out.println("Auto Scaling Group:");
System.out.println(" 1. Add new subnets to ASG configuration");
System.out.println(" 2. New instances launch in roomy subnets");
System.out.println(" 3. Scale down old instances gradually");
System.out.println();
System.out.println("EKS Cluster:");
System.out.println(" 1. Add new subnets to the cluster");
System.out.println(" 2. New pods get IPs from the big block");
System.out.println(" 3. Drain old nodes");
System.out.println();
System.out.println("RDS / Database:");
System.out.println(" 1. Add new subnets to DB subnet group");
System.out.println(" 2. Create replica or new instance in new subnet");
System.out.println(" 3. Failover");
}
}
What is the 100.64.0.0/10 pattern for EKS?
A common cause of IP exhaustion is EKS pods consuming routable IPs faster than expected. The solution is to associate a secondary CIDR from the 100.64.0.0/10 carrier-grade NAT range, keep nodes on the original routable range, and put pods on the secondary range via the VPC CNI's custom networking. This gives pods a huge private pool without consuming scarce routable space.
// EKS custom networking with secondary CIDR
package com.bank.infrastructure;
public class EksCustomNetworking {
public static void main(String[] args) {
System.out.println("Problem: EKS cluster grew, pod IPs exhausted in 10.0.0.0/16");
System.out.println();
System.out.println("Solution:");
System.out.println(" 1. Associate 100.64.0.0/16 as secondary CIDR");
System.out.println(" 2. Create subnets from 100.64.x.0/20 per AZ");
System.out.println(" 3. Configure VPC CNI custom networking");
System.out.println(" 4. Nodes keep 10.0.x.x IPs (routable)");
System.out.println(" 5. Pods get 100.64.x.x IPs (huge pool, not routable externally)");
}
}
What about peered VPCs and Transit Gateway routes?
Any peered VPC or Transit Gateway that needs to reach the new subnets must have its routes updated for the secondary CIDR. The secondary CIDR must also not overlap with any network it connects to -- the same non-overlap rule that governs the original CIDR applies to every block you add.
// Route update requirements
package com.bank.infrastructure;
public class RouteUpdateForSecondaryCidr {
public static void main(String[] args) {
System.out.println("After associating secondary CIDR 100.64.0.0/16:");
System.out.println();
System.out.println("Peered VPCs:");
System.out.println(" Add route: 100.64.0.0/16 -> pcx-peer (peering connection)");
System.out.println();
System.out.println("Transit Gateway:");
System.out.println(" Propagation should pick up the new CIDR automatically");
System.out.println(" Verify: static routes may need manual addition");
System.out.println();
System.out.println("Non-overlap: 100.64.0.0/16 must not clash with any connected network.");
}
}
What are common mistakes?
The first mistake is not planning CIDR space upfront, using a /24 for a subnet that will grow. The second is assuming the secondary CIDR works automatically with peered networks -- routes must be explicitly updated. The third is choosing a secondary CIDR that overlaps with an on-premises network or another VPC.
// Common CIDR planning mistakes
package com.bank.infrastructure;
public class CidrPlanningMistakes {
public static void main(String[] args) {
System.out.println("WRONG: /24 for a subnet that will scale heavily");
System.out.println(" 251 usable IPs exhausted in weeks");
System.out.println();
System.out.println("WRONG: secondary CIDR overlaps with on-prem 100.64.0.0/10");
System.out.println(" Routing conflicts break connectivity");
System.out.println();
System.out.println("CORRECT: plan large subnets (/20 or bigger) from the start");
System.out.println(" Check overlap with ALL connected networks before associating");
}
}
What should a senior engineer know?
A senior engineer plans CIDR space generously upfront, using /20 or larger subnets for workloads expected to grow. They understand that a VPC can have up to five CIDR blocks (raisable). They use infrastructure-as-code to manage CIDR associations and subnet creation so expansions are repeatable. For EKS specifically, they configure custom networking from day one if pod density is expected to be high, rather than retrofitting it later.
What are common interview follow-up questions?
- Can you add more than one secondary CIDR block? Yes, a VPC supports up to five CIDR blocks total, and you can request a quota increase from AWS.
- Can you remove a secondary CIDR block? Yes, but only after you delete all subnets and detach all resources associated with that CIDR.
- What happens if the secondary CIDR overlaps with a peered VPC? The peering connection will reject routes for the overlapping range, and traffic will not flow.
- How do you plan CIDR allocation to avoid this problem in the first place? Use a central IP Address Management (IPAM) tool and allocate larger blocks upfront, such as /16 for production VPCs.
package com.bank.aws;
public class CidrFollowUp {
// Planning tip: allocate /16 for production VPCs upfront
// /16 gives 65,536 addresses, enough for growth
// Use AWS VPC IPAM to track allocations across accounts
public static void main(String[] args) {
System.out.println("VPC CIDR planning:");
System.out.println(" Production: /16 (65,536 IPs)");
System.out.println(" Staging: /20 (4,096 IPs)");
System.out.println(" Dev: /22 (1,024 IPs)");
}
}
Visual Overview
Expanding IP Capacity with Secondary CIDR BEFORE: subnets nearly full ┌──────────────────────────────────────────┐ │ VPC: 10.0.0.0/16 │ │ │ │ ┌─────────────┐ ┌─────────────┐ │ │ │ Subnet AZ-a │ │ Subnet AZ-b │ │ │ │ 10.0.1.0/24 │ │ 10.0.2.0/24 │ │ │ │ 248/251 used│ │ 245/251 used│ │ │ │ NEARLY FULL │ │ NEARLY FULL │ │ │ └─────────────┘ └─────────────┘ │ └──────────────────────────────────────────┘ AFTER: secondary CIDR added, new roomy subnets ┌──────────────────────────────────────────┐ │ VPC: 10.0.0.0/16 + 100.64.0.0/16 │ │ │ │ ┌─────────────┐ ┌─────────────┐ OLD │ │ │ 10.0.1.0/24 │ │ 10.0.2.0/24 │ (drain)│ │ └─────────────┘ └─────────────┘ │ │ │ │ ┌──────────────┐ ┌──────────────┐ NEW │ │ │100.64.0.0/20 │ │100.64.16.0/20│(fill)│ │ │ 4091 IPs │ │ 4091 IPs │ │ │ │ PLENTY │ │ PLENTY │ │ │ └──────────────┘ └──────────────┘ │ └──────────────────────────────────────────┘ Migration: add new subnets to ASG/EKS -> new instances/pods launch there -> drain old subnets -> zero downtime
Key Takeaway
When a subnet runs out of IP addresses, the instinct is to make it bigger, but AWS does not allow resizing a subnet CIDR in place. The range is fixed at creation and in use -- resources have network interfaces with IPs from it, and the adjacent space may belong to another subnet. Expanding would mean renumbering live interfaces, which is disruptive and ambiguous, so AWS makes subnet CIDRs immutable.
The supported way to add capacity is at the VPC level. I associate an additional secondary CIDR with the existing VPC, create new, larger subnets from that block in each Availability Zone, and migrate workloads gradually. For auto-scaling groups I add the new subnets and let new instances launch there. For EKS I add the new subnets so new pods get IPs from the roomy block. For databases I bring up a replica in the new subnet and fail over. At no point do I renumber a running interface.
The common EKS-specific pattern is associating a secondary CIDR from the 100.64.0.0/10 carrier-grade NAT range and using the VPC CNI's custom networking to put pods on that pool while keeping nodes on the original routable range. This gives pods a huge address space without burning scarce routable IPs.
At National Bank of India our settlement EKS cluster grew faster than planned and pod IPs ran out. We associated a 100.64.0.0/16 secondary CIDR, configured custom networking, and pods started getting IPs from the new pool with no disruption. The two senior points are that peered VPCs and Transit Gateways need route updates for the new CIDR, and the secondary CIDR must not overlap with anything it connects to.
Remember one sentence: you cannot resize a subnet CIDR in place, so add a secondary CIDR to the VPC, create new subnets from it, and migrate workloads gradually without downtime.
---