Skip to content

Day 2 Operations

This is Part 2 of a multi-part, self-paced quick start exercise that will focus on day-two operations on your newly created cluster in your PaaS environment in AWS using Terraform.

What Will You Do

In part 2, you will:

  • Create a new nodegroup
  • Scale the newly created nodegroup
  • Upgrade the K8s version of your K8s cluster and nodegroups

Step 1: Configure & Provision a Nodegroup

  • Edit the terraform.tfvars file. The file location is terraform/pas_terraform/eks/terraform.tfvars. We will add a new nodegroup to the managed_nodegroups section. The managed_nodegroups section should look like the following once the new nodegroup has been added.
managed_nodegroups = {
  "ng-1" = {
    ng_name         = "infra-ng-terraform"
    node_count      = 1
    node_max_count  = 3
    node_min_count  = 1
    k8s_version     = "1.25"
    instance_type   = "t3.large"
    taint_key       = "node/infra"
    taint_operator  = "Exists"
    taint_effect    = "NoSchedule"
  },
  "ng-2" = {
    ng_name         = "worker-ng-terraform"
    node_count      = 1
    node_max_count  = 3
    node_min_count  = 1
    k8s_version     = "1.25"
    instance_type   = "t3.large"
    taint_key       = "node/worker"
    taint_operator  = "Exists"
    taint_effect    = "NoSchedule"
  }
}
  • Open the terminal or command line.
  • Navigate to the terraform/pas_terraform/eks folder.
  • Run terraform apply. Enter yes when prompted.
  • The nodegroup will be added to cluster and will be available within 10 minutes.

Step 2: Verify Nodegroup Provisioning

Once provisioning of the nodegroup is complete, you should have a new nodegroup with an additional node.

  • Navigate to the node tab for the cluster. You should see a newly added node running in the newly added nodegroup.

New Nodegroup

Step 3: Scale Nodegroup

We will now add an additional node to the nodegroup.

  • Edit the terraform.tfvars file. The file location is terraform/pas_terraform/terraform.tfvars. We will add an additional node to the managed_nodegroups section. Change the node_count to 2.
managed_nodegroups = {
  "ng-1" = {
    ng_name         = "infra-ng-terraform"
    node_count      = 1
    node_max_count  = 3
    node_min_count  = 1
    k8s_version     = "1.25"
    instance_type   = "t3.large"
    taint_key       = "node/infra"
    taint_operator  = "Exists"
    taint_effect    = "NoSchedule"
  },
  "ng-2" = {
    ng_name         = "worker-ng-terraform"
    node_count      = 2
    node_max_count  = 3
    node_min_count  = 1
    k8s_version     = "1.25"
    instance_type   = "t3.large"
    taint_key       = "node/worker"
    taint_operator  = "Exists"
    taint_effect    = "NoSchedule"
  }
}
  • Open the terminal or command line.
  • Navigate to the terraform/pas_terraform/eks folder.
  • Run terraform apply. Enter yes when prompted.
  • The new node will be added to cluster and will be available within 10 minutes.

Step 4: Verify Node Provisioning

Once provisioning of the node is complete, you should have a third node that has been added to the cluster.

  • Navigate to the node tab for the cluster. You should see a third newly added node.

New Nodegroup

Step 5: Perform Cluster Upgrade

We will now upgrade the K8s version of the control plane and nodes to a later release.

  • Edit the terraform.tfvars file. The file location is terraform/pas_terraform/eks/terraform.tfvars. We will update the k8s version for the control plane and nodegroups to a later relase. For this exercise we started at 1.25 and are upgrading to 1.26.
# Poject name variable
project               = "terraform-test-8-24-4"

# Cloud Credentials specific variables
cloud_credentials_name  = "rafay-cloud-credential"
# Specify Role ARN & externalid info below for EKS.
rolearn                 = "arn:aws:iam::679196758854:role/dreta-full-iam"
externalid              = "015e-1834-796b-71af-a5ca"

# Cluster variables ()
cluster_name           =  "terraform-test-gs-8-24-4"
# Cluster Location
cluster_location       =  "us-west-2"
# K8S Version
k8s_version            =  "1.26"

# Systems Components Placement
# Daemonset Overrides
ds_tol_key             = "cluster-node"
ds_tol_operator        = "Exists"
ds_tol_effect          = "NoSchedule"

# EKS Nodegroups
managed_nodegroups = {
  "ng-1" = {
    ng_name         = "infra-terraform"
    node_count      = 1
    node_max_count  = 3
    node_min_count  = 1
    k8s_version     = "1.26"
    instance_type   = "t3.large"
    taint_key       = "node/infra"
    taint_operator  = "Exists"
    taint_effect    = "NoSchedule"
  },
  "ng-2" = {
    ng_name         = "worker-ng-terraform"
    node_count      = 2
    node_max_count  = 3
    node_min_count  = 1
    k8s_version     = "1.26"
    instance_type   = "t3.large"
    taint_key       = "node/worker"
    taint_operator  = "Exists"
    taint_effect    = "NoSchedule"
  }
}

# TAGS
cluster_tags           = {
    "email" = "[email protected]"
    "env"    = "dev"
    "orchestrator" = "rafay"
}
node_tags = {
    "env" = "dev"
}
node_labels = {
    "app" = "infra"
    "dedicated" = "true"
}

# Blueprint/Addons specific variables
blueprint_name         = "custom-blueprint"
blueprint_version      = "v0"
base_blueprint         = "minimal"
base_blueprint_version = "1.27.0"
namespaces              = ["ingress-nginx", "cert-manager"]
infra_addons = {
    "addon1" = {
         name          = "cert-manager"
         namespace     = "cert-manager"
         addon_version = "v1.9.1"
         chart_name    = "cert-manager"
         chart_version = "v1.12.3"
         repository    = "cert-manager"
         file_path     = "file://../artifacts/cert-manager/custom_values.yaml"
         depends_on    = []
    }
    "addon2" = {
         name          = "ingress-nginx"
         namespace     = "ingress-nginx"
         addon_version = "v1.3.1"
         chart_name    = "ingress-nginx"
         chart_version = "4.2.5"
         repository    = "nginx-controller"
         file_path     = null
         depends_on    = ["cert-manager"]
    }
}

# Repository specific variables
public_repositories = {
    "nginx-controller" = {
        type = "Helm"
        endpoint = "https://kubernetes.github.io/ingress-nginx"
    }
    "cert-manager" = {
        type = "Helm"
        endpoint = "https://charts.jetstack.io"
    }
}

# Override config
overrides_config = {
    "ingress-nginx" = {
      override_addon_name = "ingress-nginx"
      override_values = <<-EOT
      controller:
        tolerations:
        - key: node/infra
          operator: Exists
          effect: NoSchedule
        - key: cluster-node
          operator: Exists
          effect: NoSchedule

        service:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: "nlb"

        admissionWebhooks:
          patch:
            tolerations:
            - key: node/infra
              operator: Exists
              effect: NoSchedule
            - key: cluster-node
              operator: Exists
              effect: NoSchedule
            # -- Labels to be added to patch job resources

      defaultBackend:
        tolerations:
        - key: node/infra
          operator: Exists
          effect: NoSchedule
        - key: cluster-node
          operator: Exists
          effect: NoSchedule
      EOT
    },
    "cert-manager" = {
      override_addon_name = "cert-manager"
      override_values = <<-EOT
      tolerations:
      - key: node/infra
        operator: Exists
        effect: NoSchedule
      - key: cluster-node
        operator: Exists
        effect: NoSchedule

      webhook:
        tolerations:
        - key: node/infra
          operator: Exists
          effect: NoSchedule
        - key: cluster-node
          operator: Exists
          effect: NoSchedule

      cainjector:
        tolerations:
        - key: node/infra
          operator: Exists
          effect: NoSchedule
        - key: cluster-node
          operator: Exists
          effect: NoSchedule

      startupapicheck:
        tolerations:
        - key: node/infra
          operator: Exists
          effect: NoSchedule
        - key: cluster-node
          operator: Exists
          effect: NoSchedule
      EOT
    }
}
  • Open the terminal or command line.
  • Navigate to the terraform/pas_terraform/eks folder.
  • Run terraform apply. Enter yes when prompted.
  • The cluster should now show that it is upgrading.

Cluster Upgrading

  • The upgrade process takes about 30-40 minutes to complete.

Step 6: Verify Cluster Upgrade

Once the cluster upgrade is complete, your control plane and nodes should be on a later release.

  • Navigate to the Clusters Upgrade Jobs tab. You should see that the upgrade job has completed and the cluster is now running the later K8s version.

Upgrade Complete

Recap

Congratulations! At this point, you have

  • Successfully configured and provisioned a new nodegroup for your EKS cluster
  • Added a second node to your newly created nodegroup
  • Upgraded your cluster's control plane and nodegroups to a later K8s version