Skip to content

Part 3: Blueprint

What Will You Do

In this part of the self-paced exercise, you will create a custom cluster blueprint with a Karpenter add-on, based on declarative specifications.

Step 1: Create Repository

In this step, you will create a repository in your project so that the controller can retrieve the Helm charts automatically.

  • Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you forked the Git repository
  • Navigate to the folder "/getstarted/karpenter/repository"

The "repository.yaml" file contains the declarative specification for the repository. In this case, the specification is of type "Helm Repository" and the "endpoint" is pointing to the Karpenter Helm chart. 

apiVersion: config.rafay.dev/v2
kind: Repository
metadata:
  name: karpenter-repo
spec:
  repositoryType: HelmRepository
  endpoint:  https://charts.karpenter.sh
  credentialType: CredentialTypeNotSet

Type the command below 

rctl create repository -f repository.yaml

If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.

  • Navigate to the "defaultproject" project in your Org
  • Select Integrations -> Repositories

Repository

Step 2: Create Namespace

In this step, you will create a namespace for Karpenter. The "namespace.yaml" file contains the declarative specification

The following items may need to be updated/customized if you made changes to these or used alternate names.

  • value: karpenter-cluster
kind: ManagedNamespace
apiVersion: config.rafay.dev/v2
metadata:
  name: karpenter
  description: namespace for karpenter
  labels:
  annotations:
spec:
  type: RafayWizard
  resourceQuota:
  placement:
    placementType: ClusterSpecific
    clusterLabels:
    - key: rafay.dev/clusterName
      value: karpenter-cluster
  • Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you forked the Git repository
  • Navigate to the folder "/getstarted/karpenter/namespace"
  • Type the command below 
rctl create namespace -f namespace.yaml

If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.

  • Navigate to the "defaultproject" project in your Org
  • Select Infrastructure -> Namespaces
  • You should see an namesapce called "karpenter"

Namespace

Step 3: Create Addons

In this step, you will create two custom addons, one for Karpenter and a second for the Karpenter Provisioner.

First, we must update the "custom-values.yaml" file for the Karpenter addon with the cluster endpoint.

  • Navigate to the "defaultproject" project in your Org
  • Select Infrastructure -> Clusters
  • Select "karpenter-cluster" -> Configuration -> "View More"
  • Copy the "Endpoint" value and update the "custom-values.yaml" file with it

Karpenter Addon

serviceAccount:
  create: false
controller:
  clusterName: karpenter-cluster
  clusterEndpoint: https://F542D3537D0D5FC326A1FE6FD1368640.gr7.us-west-2.eks.amazonaws.com

The following details are used to build the addon declarative specification.

  • "v1" because this is our first version
  • The addon is part of the "defaultproject"
  • Name of addon is "karpenter-addon"
  • The addon will be deployed to a namespace called "karpenter"
  • You will be using a "custom-values.yaml" as an override which is located in the folder "/getstarted/karpenter/addon"
  • The "karpenter" chart will be used from the previously created repository named "karpenter-repo"

The following items may need to be updated/customized if you made changes to these or used alternate names.

  • namespace: "karpenter"
  • repository_ref: "karpenter-repo"
kind: AddonVersion
metadata:
  name: v1
  project: defaultproject
spec:
  addon: karpenter-addon
  namespace: karpenter
  template:
    type: Helm3
    valuesFile: custom-values.yaml
    repository_ref: karpenter-repo
    repo_artifact_meta:
      helm:
       chartName: karpenter
  • Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you forked the Git repository
  • Navigate to the folder "/getstarted/karpenter/addon"
  • Type the command below 
rctl create addon version -f karpenter-addon.yaml

If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.

  • Navigate to the "defaultproject" project in your Org
  • Select Infrastructure -> Addons
  • You should see an addon called "karpenter-addon"

Addon

Next, we will create the second custom addon for the Karptenter Provisioner.

First, we must update the "provisioner.yaml" file with the Instance Profile for the cluster. To obtain the Instance Profile, we can use the AWS CLI.

  • Navigate to the "defaultproject" project in your Org
  • Select Infrastructure -> Clusters
  • Select "karpenter-cluster" -> Node Groups
  • Copy the Node Instance Role name from the Node Instance Role ARN

Addon

  • Execute the below command after updating the "role-name" and "region" to match your environment
aws iam list-instance-profiles-for-role --role-name rafay-karpenter-cluster-nodegroup-NodeInstanceRole-1VOXQFP8RFNS3 --region us-west-2 --query InstanceProfiles[].InstanceProfileName
  • Copy the returned Instance Profile and update the "provisioner.yaml" file with the Instance Profile value.

Note that the "cluster-name" is set to match the name of the cluster and the AWS tags that were specified in the cluster spec file.

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  requirements:
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["spot"]
  limits:
    resources:
      cpu: 1000
  provider:
    instanceProfile: eks-f2bee252-cc24-d685-b6f5-9d4a917f2724
    securityGroupSelector:
      cluster-name: karpenter-cluster
    subnetSelector:
      cluster-name: karpenter-cluster
  ttlSecondsAfterEmpty: 30

The following details are used to build the provisioner addon declarative specification.

  • "v1" because this is our first version
  • The addon is part of the "defaultproject"
  • Name of addon is "provisioner-addon"
  • The addon will be deployed to a namespace called "karpenter"
  • You will be using the "provisioner.yaml" which is located in the folder "/getstarted/karpenter/addon"

The following items may need to be updated/customized if you made changes to these or used alternate names.

  • namespace: "karpenter"
kind: AddonVersion
metadata:
  name: v1
  project: defaultproject
spec:
  addon: provisioner-addon
  namespace: karpenter
  template:
    type: NativeYaml
    yamlFile: provisioner.yaml
  • Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you forked the Git repository
  • Navigate to the folder "/getstarted/karpenter/addon"
  • Type the command below 
rctl create addon version -f provisioner-addon.yaml

If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.

  • Navigate to the "defaultproject" project in your Org
  • Select Infrastructure -> Addons
  • You should see an addon called "provisioner-addon"

Addon

Step 4: Create Blueprint

In this step, you will create a custom cluster blueprint with the Karpenter addon and the provisioner addon. The "blueprint.yaml" file contains the declarative specification.

  • Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you forked the Git repository
  • Navigate to the folder "/getstarted/karpenter/blueprint"

The following items may need to be updated/customized if you made changes to these or used alternate names.

  • project: "defaultproject"
kind: Blueprint
metadata:
  # blueprint name
  name: karpenter-blueprint
  #project name
  project: defaultproject
  • Type the command below 
rctl create blueprint -f blueprint.yaml

If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.

  • Navigate to the "defaultproject" project in your Org
  • Select Infrastructure -> Blueprint
  • You should see an blueprint called "karpenter-blueprint

Karpenter Blueprint

New Version

Although we have a custom blueprint, we have not provided any details on what it comprises. In this step, you will create and add a new version to the custom blueprint. The YAML below is a declarative spec for the new version.

The following items may need to be updated/customized if you made changes to these or used alternate names.

  • project: "defaultproject"
  • blueprint: "karpenter-blueprint"
  • name: "karpenter-addon"
  • version: "v1"
  • name: "provisioner-addon"
  • version: "v1"
kind: BlueprintVersion
metadata:
  name: v1
  project: defaultproject
  description: Karpenter
spec:
  blueprint: karpenter-blueprint
  baseSystemBlueprint: default
  baseSystemBlueprintVersion: ""
  addons:
    - name: karpenter-addon
      version: v1
    - name: provisioner-addon
      version: v1
  # cluster-scoped or namespace-scoped
  pspScope: cluster-scoped
  rafayIngress: true
  rafayMonitoringAndAlerting: true
  kubevirt: false
  # BlockAndNotify or DetectAndNotify
  driftAction: BlockAndNotify
  • Type the command below to add a new version
rctl create blueprint version -f blueprint-v1.yaml

If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.

  • Navigate to the "defaultproject" project in your Org
  • Select Infrastructure -> Blueprint
  • Click on the "karpenter-blueprint" custom cluster blueprint

v1 Karpenter Blueprint

Step 5: Update Cluster Blueprint

In this step, you will update the cluster to use the previously created custom blueprint with the Karpenter addon and the Provisioner addon.

  • Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you forked the Git repository
  • Navigate to the folder "/getstarted/karpenter/blueprint"
  • Type the command below 
rctl apply -f cluster-update.yaml

If there are no errors, you will be presented with a "Task ID" that you can use to check progress/status. 

Cluster: karpenter-cluster
{
  "taskset_id": "7kr93wk",
  "operations": [
    {
      "operation": "BlueprintUpdation",
      "resource_name": "karpenter-cluster",
      "status": "PROVISION_TASK_STATUS_PENDING"
    }
  ],
  "comments": "The status of the operations can be fetched using taskset_id",
  "status": "PROVISION_TASKSET_STATUS_PENDING"
}

If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.

  • Navigate to the "defaultproject" project in your Org
  • Select Infrastructure -> Clusters
  • You should see the cluster is now using the "karpenter-blueprint

Karpenter Blueprint

  • Navigate to Infrastructure -> Clusters
  • Click on "KUBECTL" in the "karpenter-cluster" cluster card
  • Type the command below 
kubectl get pods --namespace karpenter
  • You should see a result like the following.
NAME                                   READY   STATUS    RESTARTS   AGE
karpenter-controller-f8b66f884-r6s84   1/1     Running   0          16m
karpenter-webhook-588f965b4d-v8pxp     1/1     Running   0          16m

Recap

As of this step, you have created a "cluster blueprint" with Karpenter and the Karpenter Provisioner as addons, and applied this blueprint to the existing cluster. You are now ready to move onto the next step where you will deploy a test workload to scale the cluster with Karpenter.