KOP Recipes - Hashicorp Vault

Properly securing application secrets is one of the most neglected methods in the industry today. HashiCorp's Vault is a very popular Secrets Manager solution that can be used to secure, store and tightly control access to tokens, passwords, certificates, encryption keys.

This recipe is geared towards users that would like to deploy a Vault server in Kubernetes for testing purposes. Please follow HashiCorp's best practices to deploy and operate Vault Server for production.

What Will You Do¶

In this exercise,

You will create a workload using HashiCorp Vault's official Helm chart
You will then deploy Vault to a managed cluster (perhaps hosting shared infrastructure services used by applications deployed in the same or different clusters)

Important

This recipe describes the steps to create and use a Vault workload using the Web Console. The entire workflow can also be fully automated and embedded into an automation pipeline.

Assumptions¶

You have already provisioned or imported a Kubernetes cluster using the controller
You have Helm CLI installed locally to download the Vault helm chart

Challenges¶

Although deploying a simple Helm chart can be trivial for a quick sniff test, there are a number of considerations that have to be factored in for a stable deployment. Some of them are described below.

Ingress¶

The Vault service deployed on the cluster needs to be exposed externally for it to be practical. In this recipe, we will use the managed nginx Ingress Controller in the default blueprint to expose the Vault service externally.

Certificate Lifecycle¶

Vault's Ingress needs to be secured using TLS. It is impractical to manually handle certificates and private keys. In this recipe, we will use a managed cert-manager addon in our cluster blueprint to manage the lifecycle of certificates for the Vault Server's Ingress.

Using Vault¶

Integrating Vault with your applications can be very complex and cumbersome. We have developed a deep integration with Vault that makes it extremely easy and intuitive for developers and operations teams to leverage their Vault server. Just add a few annotations to your existing YAML or Helm charts to transparently leverage Vault.

Step 1: Download Helm chart¶

Use your helm client to download the latest release of Vault helm chart file vault-x.y.z.tgz to your machine vault-helm-chart

Add HashiCorp's repo to your Helm CLI

helm repo add hashicorp https://helm.releases.hashicorp.com

Now, fetch the latest Helm chart for Vault from this repo.

helm fetch hashicorp/vault

Note

In this recipe, we used v0.6.0 of the Vault Helm chart

Step 2: Customize Values¶

In this step, we will be creating a custom "values.yaml" file with overrides for our Vault Server deployment. Copy the following yaml document into the "vault-custom-values.yaml" file.

Copy the following yaml document into the "vault-custom-values.yaml" file

## Disable vault agent injection
##
injector:
  enabled: false

server:
  ## Define resource limit and limit for your Vault pod
  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 256Mi
      cpu: 250m
  ## Configure ingress to expose Vault service externally
  ingress:
    enabled: true
    annotations:
      ## Add annotation to use the built-in nginx ingress controller
      kubernetes.io/ingress.class: nginx
      ## Add annotation to use cert-manager for generating and maintaining the cert for Vault Server ingress
      cert-manager.io/cluster-issuer: "letsencrypt-http"
    hosts:
      ## Change the host to your domain
      - host: vault.infra.gorafay.net
        paths: ["/"]
    tls:
      - secretName: vault-ingress-tls
        hosts:
          - vault.infra.gorafay.net
  ## Disable readinessProbe and livenessProbe so Vault pod can be running after initial deployment to do the vault init and unseal process
  readinessProbe:
    enabled: false
  livenessProbe:
    enabled: false
  dataStorage:
    enabled: true
    ## Set the size of the data storage for Vault server
    size: 10Gi
  auditStorage:
    enabled: true
    ## Set the size of the audit data storage for Vault server
    size: 10Gi
## Enable Vault UI
ui:
  enabled: true

Step 3: Create Workload¶

Login into the Web Console and navigate to your Project as an Org Admin or Project Admin
Under Infrastructure (or Applications if accessed with Project Admin role), select "Namespaces" and create a new namespace called "vault"
Go to Applications > Workloads
Select "New Workload" to create a new workload called "vault"
Ensure that you select "Helm" for Package Type and select the namespace as "vault"
Click CONTINUE to next step

Create Vault Workload

Upload the downloaded Vault helm chart vault-x.y.z.tgz to the Helm > Choose File
Upload the vault-custom-values.yaml created file from the previous step to Values.yaml > Choose File
Save and Go to Placement for the next step

Create vault workload

Select the cluster that you would like to deploy Vault
Publish the Vault workload to the selected cluster

Step 4: Verify Deployment¶

You can optionally verify whether the correct resources have been created on the cluster.

Once the workload is published, click on Debug
Click on Kubectl to open a virtual terminal for kubectl proxy access right to the "vault" namespace context of the cluster

vault workload debug

First, we will verify the status of the pods

kubectl get pod

- Next, we will verify the Vault Server's persistent volume claim status

kubectl get pvc

Next, we will verify the Ingress for Vault service

kubectl get ingress

Finally, we will verify the Vault service

kubectl get svc

Shown below is an example for what you should see on a cluster where Vault has been deployed as a Helm type workload through the controller.

Verify Deployment

Alternatively, users with Infrastructure Admin or Organization Admin roles can view the status of all Kubernetes resources created by this Vault workload by going to Infrastructure > Clusters > cluster_name > Resources and filter by Workloads "vault" as below:

Verify Deployment

Step 5: Initialize Vault¶

After the Vault workload is published successfully to the cluster, access the Vault service via the ingress hostname to initialize and unseal the Vault server. The Vault server can do its job only after this operation is performed.

Access the Vault ingress hostname and enter the desired number of key shares (for e.x. 5) and key threshold (for e.x. 3) to start initializing the Vault server

vault init

After the Vault server is initialized, download the key file and store it securely

vault init

Start unsealing the Vault server by entering the keys from the "keys_base64" of the downloaded key file. In the example above with key threshold number set to 3 and key shares set to 5, select 3 out of 5 keys in the "keys_base64" object to unseal the Vault server

vault unseal

After the Vault server is unsealed, login to Vault server using the root token from the "root_token" of the downloaded key file

vault login

You should be successfully logged in to Vault server and the server status should report Green

vault done

Recap¶

Congratulations! You have successfully deployed the Vault server to a managed cluster. You also initialized and unsealed the Vault server.

You can now start using this Vault Server workload to secure application secrets. More information on Vault integration with the controller can be found at here