Part 3: Blueprint
What Will You Do¶
In this part of the self-paced exercise, you will create a custom cluster blueprint with a Karpenter add-on, based on declarative specifications.
Step 1: Create Repository¶
In this step, you will create a repository in your project so that the controller can retrieve the Helm charts automatically.
- Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you forked the Git repository
- Navigate to the folder "
/getstarted/karpenter/repository"
The "repository.yaml" file contains the declarative specification for the repository. In this case, the specification is of type "Helm Repository" and the "endpoint" is pointing to the Karpenter Helm chart.
apiVersion: config.rafay.dev/v2
kind: Repository
metadata:
name: karpenter-repo
spec:
repositoryType: HelmRepository
endpoint: https://charts.karpenter.sh
credentialType: CredentialTypeNotSet
Type the command below
rctl create repository -f repository.yaml
If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.
- Navigate to the "defaultproject" project in your Org
- Select Integrations -> Repositories
Step 2: Create Namespace¶
In this step, you will create a namespace for Karpenter. The "namespace.yaml" file contains the declarative specification
The following items may need to be updated/customized if you made changes to these or used alternate names.
- value: karpenter-cluster
kind: ManagedNamespace
apiVersion: config.rafay.dev/v2
metadata:
name: karpenter
description: namespace for karpenter
labels:
annotations:
spec:
type: RafayWizard
resourceQuota:
placement:
placementType: ClusterSpecific
clusterLabels:
- key: rafay.dev/clusterName
value: karpenter-cluster
- Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you forked the Git repository
- Navigate to the folder "
/getstarted/karpenter/namespace" - Type the command below
rctl create namespace -f namespace.yaml
If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.
- Navigate to the "defaultproject" project in your Org
- Select Infrastructure -> Namespaces
- You should see an namesapce called "karpenter"
Step 3: Create Addons¶
In this step, you will create two custom addons, one for Karpenter and a second for the Karpenter Provisioner.
First, we must update the "custom-values.yaml" file for the Karpenter addon with the cluster endpoint.
- Navigate to the "defaultproject" project in your Org
- Select Infrastructure -> Clusters
- Select "karpenter-cluster" -> Configuration -> "View More"
- Copy the "Endpoint" value and update the "custom-values.yaml" file with it
serviceAccount:
create: false
controller:
clusterName: karpenter-cluster
clusterEndpoint: https://F542D3537D0D5FC326A1FE6FD1368640.gr7.us-west-2.eks.amazonaws.com
The following details are used to build the addon declarative specification.
- "v1" because this is our first version
- The addon is part of the "defaultproject"
- Name of addon is "karpenter-addon"
- The addon will be deployed to a namespace called "karpenter"
- You will be using a "custom-values.yaml" as an override which is located in the folder "
/getstarted/karpenter/addon" - The "karpenter" chart will be used from the previously created repository named "karpenter-repo"
The following items may need to be updated/customized if you made changes to these or used alternate names.
- namespace: "karpenter"
- repository_ref: "karpenter-repo"
kind: AddonVersion
metadata:
name: v1
project: defaultproject
spec:
addon: karpenter-addon
namespace: karpenter
template:
type: Helm3
valuesFile: custom-values.yaml
repository_ref: karpenter-repo
repo_artifact_meta:
helm:
chartName: karpenter
- Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you forked the Git repository
- Navigate to the folder "
/getstarted/karpenter/addon" - Type the command below
rctl create addon version -f karpenter-addon.yaml
If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.
- Navigate to the "defaultproject" project in your Org
- Select Infrastructure -> Addons
- You should see an addon called "karpenter-addon"
Next, we will create the second custom addon for the Karptenter Provisioner.
First, we must update the "provisioner.yaml" file with the Instance Profile for the cluster. To obtain the Instance Profile, we can use the AWS CLI.
- Navigate to the "defaultproject" project in your Org
- Select Infrastructure -> Clusters
- Select "karpenter-cluster" -> Node Groups
- Copy the Node Instance Role name from the Node Instance Role ARN
- Execute the below command after updating the "role-name" and "region" to match your environment
aws iam list-instance-profiles-for-role --role-name rafay-karpenter-cluster-nodegroup-NodeInstanceRole-1VOXQFP8RFNS3 --region us-west-2 --query InstanceProfiles[].InstanceProfileName
- Copy the returned Instance Profile and update the "provisioner.yaml" file with the Instance Profile value.
Note that the "cluster-name" is set to match the name of the cluster and the AWS tags that were specified in the cluster spec file.
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: default
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot"]
limits:
resources:
cpu: 1000
provider:
instanceProfile: eks-f2bee252-cc24-d685-b6f5-9d4a917f2724
securityGroupSelector:
cluster-name: karpenter-cluster
subnetSelector:
cluster-name: karpenter-cluster
ttlSecondsAfterEmpty: 30
The following details are used to build the provisioner addon declarative specification.
- "v1" because this is our first version
- The addon is part of the "defaultproject"
- Name of addon is "provisioner-addon"
- The addon will be deployed to a namespace called "karpenter"
- You will be using the "provisioner.yaml" which is located in the folder "
/getstarted/karpenter/addon"
The following items may need to be updated/customized if you made changes to these or used alternate names.
- namespace: "karpenter"
kind: AddonVersion
metadata:
name: v1
project: defaultproject
spec:
addon: provisioner-addon
namespace: karpenter
template:
type: NativeYaml
yamlFile: provisioner.yaml
- Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you forked the Git repository
- Navigate to the folder "
/getstarted/karpenter/addon" - Type the command below
rctl create addon version -f provisioner-addon.yaml
If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.
- Navigate to the "defaultproject" project in your Org
- Select Infrastructure -> Addons
- You should see an addon called "provisioner-addon"
Step 4: Create Blueprint¶
In this step, you will create a custom cluster blueprint with the Karpenter addon and the provisioner addon. The "blueprint.yaml" file contains the declarative specification.
- Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you forked the Git repository
- Navigate to the folder "
/getstarted/karpenter/blueprint"
The following items may need to be updated/customized if you made changes to these or used alternate names.
- project: "defaultproject"
kind: Blueprint
metadata:
# blueprint name
name: karpenter-blueprint
#project name
project: defaultproject
- Type the command below
rctl create blueprint -f blueprint.yaml
If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.
- Navigate to the "defaultproject" project in your Org
- Select Infrastructure -> Blueprint
- You should see an blueprint called "karpenter-blueprint
New Version¶
Although we have a custom blueprint, we have not provided any details on what it comprises. In this step, you will create and add a new version to the custom blueprint. The YAML below is a declarative spec for the new version.
The following items may need to be updated/customized if you made changes to these or used alternate names.
- project: "defaultproject"
- blueprint: "karpenter-blueprint"
- name: "karpenter-addon"
- version: "v1"
- name: "provisioner-addon"
- version: "v1"
kind: BlueprintVersion
metadata:
name: v1
project: defaultproject
description: Karpenter
spec:
blueprint: karpenter-blueprint
baseSystemBlueprint: default
baseSystemBlueprintVersion: ""
addons:
- name: karpenter-addon
version: v1
- name: provisioner-addon
version: v1
# cluster-scoped or namespace-scoped
pspScope: cluster-scoped
rafayIngress: true
rafayMonitoringAndAlerting: true
kubevirt: false
# BlockAndNotify or DetectAndNotify
driftAction: BlockAndNotify
- Type the command below to add a new version
rctl create blueprint version -f blueprint-v1.yaml
If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.
- Navigate to the "defaultproject" project in your Org
- Select Infrastructure -> Blueprint
- Click on the "karpenter-blueprint" custom cluster blueprint
Step 5: Update Cluster Blueprint¶
In this step, you will update the cluster to use the previously created custom blueprint with the Karpenter addon and the Provisioner addon.
- Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you forked the Git repository
- Navigate to the folder "
/getstarted/karpenter/blueprint" - Type the command below
rctl apply -f cluster-update.yaml
If there are no errors, you will be presented with a "Task ID" that you can use to check progress/status.
Cluster: karpenter-cluster
{
"taskset_id": "7kr93wk",
"operations": [
{
"operation": "BlueprintUpdation",
"resource_name": "karpenter-cluster",
"status": "PROVISION_TASK_STATUS_PENDING"
}
],
"comments": "The status of the operations can be fetched using taskset_id",
"status": "PROVISION_TASKSET_STATUS_PENDING"
}
If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.
- Navigate to the "defaultproject" project in your Org
- Select Infrastructure -> Clusters
- You should see the cluster is now using the "karpenter-blueprint
- Navigate to Infrastructure -> Clusters
- Click on "KUBECTL" in the "karpenter-cluster" cluster card
- Type the command below
kubectl get pods --namespace karpenter
- You should see a result like the following.
NAME READY STATUS RESTARTS AGE
karpenter-controller-f8b66f884-r6s84 1/1 Running 0 16m
karpenter-webhook-588f965b4d-v8pxp 1/1 Running 0 16m
Recap¶
As of this step, you have created a "cluster blueprint" with Karpenter and the Karpenter Provisioner as addons, and applied this blueprint to the existing cluster. You are now ready to move onto the next step where you will deploy a test workload to scale the cluster with Karpenter.