Skip to content

Secondary CIDR with VPC

As customers deploy more and more workloads to their Kubernetes clusters they can quickly get into situations where pods are consuming thousands of IP Addresses. This can quickly exhaust the pool of IP Addresses allocated to the primary VPC. In scenarios where it's not possible to recreate or extend the VPC's CIDR blocks they can deploy their worker nodes, pod, and service network to a newly created larger CIDR block (e.g. 100.64.0.0/8) within the VPC. Newly created clusters can be configured to use the subnets defined in the secondary CIDR block. Existing clusters can migrate workloads over to the secondary CIDR block. This can be done by creating a new node group that utilize the subnets within the secondary CIDR block, once this is complete you can drain and delete the existing node group.


Architecture

In the described scenario it is possible to run the EKS control plane on subnets defined from the primary CIDR block while migrating the instances and pod network to subnets defined from the secondary CIDR block. In this case both the primary and secondary ENIs will reside in the VPCs secondary CIDR range. This allows the Kubernetes cluster the ability to utilize all the IP addresses allocated to all ENIs. Those IP Addresses previously reserved for the host network can now be used by the pod network increasing the node's pod density.

Secondary CIDR with VPC


VPC Configuration

Use the following VPC configuration as a reference to the content below.

Name Subnet-Id CIDR AZ
SecondarySubnetPrivateUSWEST2A subnet-08a7a7fdd3f23f18d 100.64.0.0/20 us-west-2a
SecondarySubnetPrivateUSWEST2B subnet-09629aeeae61cfd36 100.64.16.0/20 us-west-2b
SecondarySubnetPrivateUSWEST2D subnet-061adebec170d8ba3 100.64.32.0/20 us-west-2d
PrimarySubnetPrivateUSWEST2A subnet-04e3ccd47c9ef9978 192.168.96.0/19 us-west-2a
PrimarySubnetPrivateUSWEST2B subnet-06e2b1a5cc98cac91 192.168.120.0/19 us-west-2b
PrimarySubnetPrivateUSWEST2D subnet-030531e64a35bac1a 192.168.160.0/19 us-west-2d
PrimarySubnetPublicUSWEST2A subnet-016ec9f5fa892cac8 192.168.0.0/19 us-west-2a
PrimarySubnetPublicUSWEST2B subnet-02aeb278cfa1d7835 192.168.32.0/19 us-west-2b
PrimarySubnetPublicUSWEST2D subnet-001748a513e7d39e5 192.168.64.0/19 us-west-2d

Cluster Configuration

To utilize the subnets on the primary CIDR block for your control plane specify the primary subnets under the cluster settings during the cluster configuration as shown below.

Cluster Configuration VPC

Node Group Configuration

To utilize the subnets on the secondary CIDR block for your worker nodes and pod network specify the secondary subnets under the node group settings during the cluster configuration as shown below.

Node Group Configuration VPC

Sample Cluster Specification

If creating a cluster from a cluster specification you can define the subnets from the secondary CIDR block to be used for the node group under the subnets section in the node group as shown below.

kind: Cluster
metadata:
  name: my-eks-cluster
  project: my-project
spec:
  blueprint: default
  cloudprovider: my-cloud-provider
  clusterConfig: null
  cniprovider: aws-cni
  proxyconfig: {}
  type: eks
---
apiVersion: rafay.io/v1alpha5
kind: ClusterConfig
managedNodeGroups:
- amiFamily: AmazonLinux2
  desiredCapacity: 2
  iam:
    withAddonPolicies:
      albIngress: true
      autoScaler: true
      externalDNS: true
  instanceType: t3.large
  maxSize: 2
  minSize: 2
  name: my-ng-on-secondary-cidr
  privateNetworking: true
  subnets:
  - subnet-08a7a7fdd3f23f18d
  - subnet-09629aeeae61cfd36
  - subnet-061adebec170d8ba3
  volumeSize: 80
  volumeType: gp3
metadata:
  name: my-eks-cluster
  region: us-west-2
  version: "1.21"
vpc:
  clusterEndpoints:
    privateAccess: true
    publicAccess: false
  nat:
    gateway: Single
  id: vpc-09d7879cdaea507e9
  subnets:
    private:
      subnet-04e3ccd47c9ef9978:
        id: subnet-04e3ccd47c9ef9978
      subnet-06e2b1a5cc98cac91:
        id: subnet-06e2b1a5cc98cac91
      subnet-030531e64a35bac1a:
        id: subnet-030531e64a35bac1a
    public:
      subnet-02aeb278cfa1d7835:
        id: subnet-02aeb278cfa1d7835
      subnet-016ec9f5fa892cac8:
        id: subnet-016ec9f5fa892cac8
      subnet-001748a513e7d39e5:
        id: subnet-001748a513e7d39e5