Skip to content

CloudWatch

Although the Kubernetes Management Operator on managed EKS clusters provides integrated monitoring capabilities, organizations may have standardized on AWS CloudWatch for their application log aggregation and/or cluster monitoring .

CloudWatch Container Insights provides a way to collect, aggregate, correlate, and summarize metrics and logs from containers running on ECS, EKS, and linux k8s platforms running ON Amazon EC2. This recipe describes how customers can standardize the configuration, deployment, and lifecycle management of CloudWatch Container Insights across their fleet of clusters.


What Will You Do

In this exercise,

  • You will create a customized "cloudwatch-metrics" addon utilizing the cloudwatch-agent
  • You will use the addon in a custom cluster blueprint
  • You will then apply this cluster blueprint to a managed cluster

Important

This recipe describes the steps to create and use a custom cluster blueprint using the Web Console. The entire workflow can also be fully automated and embedded into an automation pipeline.


Assumptions

  • You have already provisioned or imported one or more Kubernetes clusters using the controller.
  • You have attached the IAM policy CloudWatchAgentServerPolicy to the IAM role attached to your worker nodes.

Step 1: Create K8s CloudWatch agent K8s YAML manifest

  • Download the ServiceAccount manifest
wget https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-serviceaccount.yaml
  • Download the ConfigMap manifest
wget https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-configmap.yaml
  • Edit the highlighted lines to match your cluster configuration
apiVersion: v1
data:
  # Configuration is in Json format. No matter what configure change you make,
  # please keep the Json blob valid.
  cwagentconfig.json: |
    {
      "logs": {
        "metrics_collected": {
          "kubernetes": {
            "cluster_name": "my-cluster-name",
            "metrics_collection_interval": 60
          }
        },
        "force_flush_interval": 5
      }
    }
kind: ConfigMap
metadata:
  name: cwagentconfig
  namespace: amazon-cloudwatch
  • Download the DaemonSet YAML
wget  https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-daemonset.yaml
  • Merge the three files into a new file called "cloudwatch-metrics-agent.yaml", don't forget to add "---" between each file. You can use the commands below to merge the files.
cat cwagent-configmap.yaml >> cloudwatch-metrics-agent.yaml
echo "" >> cloudwatch-metrics-agent.yaml
echo "---" >> cloudwatch-metrics-agent.yaml
cat cwagent-serviceaccount.yaml >> cloudwatch-metrics-agent.yaml
echo "" >> cloudwatch-metrics-agent.yaml
echo "---" >> cloudwatch-metrics-agent.yaml
cat cwagent-daemonset.yaml >> cloudwatch-metrics-agent.yaml

Step 2: Create Addon

  • Login into the Web Console and navigate to your Project as an Org Admin or Infrastructure Admin
  • Under Infrastructure, select "Namespaces" and create a new namespace called "amazon-cloudwatch", Set the Pod Security Policy to "rafay-privileged-psp"
  • Select "Addons" and "Create" a new Addon called "cloudwatch-metrics"
  • Ensure that you select "K8s YAML" for the type, "Upload files manually" for the Artifact Sync, and set the namespace as "amazon-cloudwatch"
  • Click on "+New Version"
  • Enter "v1.2.247347" for the Version Name and "UPLOAD" the file created in step 1.
  • Select "Save Changes"
  • Once the addon is created, publish it and optionally provide a version so that it can be tracked.

Create Addon


Step 3: Create Blueprint

Now, we are ready to assemble a custom cluster blueprint using the newly created CloudWatch addon. We can add additional addons to the blueprint at the same time.

  • Under Infrastructure, select "Blueprints"
  • Create a new blueprint and give it a name such as "standard-blueprint"
  • Set the Version Name
  • Set the PSP Policy Type to "cluster-scoped"
  • Select the ""cloudwatch-metrics" addon
  • Disable Managed System Add-On "Monitoring & Alerting"
  • Select "Save Changes"

Create Blueprint


Step 4: Apply Blueprint

Now, we are ready to apply this custom blueprint to a cluster.

  • Click on Options for the target Cluster in the Web Console
  • Select "Update Blueprint" and select the "standard-blueprint" blueprint we created from the list

Create Blueprint

  • Click on "Save and Publish".

This will start the deployment of the addons configured in the "standard-blueprint" blueprint to the targeted cluster. The blueprint sync process can take a few minutes. Once complete, the cluster will display the current cluster blueprint details and whether the sync was successful or not.


Step 5: Verify Blueprint

Users can optionally verify whether the required resources for the custom blueprint were created on the cluster. Click on the Kubectl button on the cluster to open a virtual terminal

First, we will verify if the "amazon-cloudwatch" namespace has been created

kubectl get ns amazon-cloudwatch

NAME                STATUS   AGE
amazon-cloudwatch   Active   6m17s

Next, we will verify that the required DaemonSet and pods were created in the "amazon-cloudwatch" namespace. You should see something like the example below.

kubectl get ds -n amazon-cloudwatch

NAME               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
cloudwatch-agent   2         2         2       2            2           <none>          3m53s
kubectl get pod -n amazon-cloudwatch

NAME                     READY   STATUS    RESTARTS   AGE
cloudwatch-agent-4pn8v   1/1     Running   0          3m13s
cloudwatch-agent-sbbvk   1/1     Running   0          3m13s

Step 6: View Metrics in CloudWatch

You can now verify in CloudWatch that the metrics are being collected from the cluster. The following Log group will be created and populated with their appropriate logs.

  • /aws/containerinsights/my-cluster-name/performance

View CloudWatch


Recap

Congratulations! You have successfully created a custom cluster blueprint with the "cloudwatch-metrics" addon and applied it to a cluster. You can now use this blueprint on as many clusters as you require.