AWS Integration

Overview

Cost Management retrieves asset prices from the public AWS pricing API by default. You can leverage Cost Profile to interface directly with your account to receive accurate price data from AWS. This integration will properly account for Enterprise Discount Programs, Reserved Instance usage, Savings Plans, spot usage, and more.

Necessary permissions are required to create the Cost and Usage Report (CUR), and to add IAM credentials for Athena and S3. The ability to add and use CloudFormation templates is an optional permission.

---

Users must perform the below steps and complete a few setups to create a Cost profile in the controller.

Step 1: CUR Setup

• Refer CUR Creation for creating Cost and Usage Reports. It is mandatory to enable Resource IDs and Athena integration when creating the CUR. Note down the bucket name that you create for CUR data

• If the Billing and Cost Management page cannot be accessed, the owner of the organization’s root account must follow the instructions provided here

---

Step 2: Athena Setup

In addition, as a part of CUR creation, Amazon creates a CloudFormation template that is used to create the Athena integration. This template is created in the CUR S3 bucket under user-billing-prefix/curname and typically has the filename crawler-cfn.yml. Deploy this CloudFormation template to complete the CUR Athena integration. Refer [here]https://docs.aws.amazon.com/cur/latest/userguide/use-athena-cf.html to know more details on CloudFormation templates.

Once the Athena setup is complete with the CUR, create a new S3 bucket for Athena query results.

• Navigate to AWS S3 and select Create Bucket
• Choose a name that adheres to the format aws-athena-query-results-* and make sure to use the same region as the one that was used for the CUR bucket
• Select Create Bucket
• Navigate to AWS Athena and select Settings
• Set Query result location to the S3 bucket that got created now

---

Step 3: IAM Permissions Setup

Users can add IAM Permission in two ways, via CloudFormation or Manually.

Via CloudFormation

Cost Management offers a set of CloudFormation templates to help set your IAM roles up. New users are suggested to download the templates and provision IAM roles. Download the template files from the below provided URLs and upload them as the stack template over here

Kubernetes clusters run in the same account as the master payer account

Perform the following steps:

• Download this file

• Navigate to CloudFormation

• Click Create New Stack if not used AWS CloudFormation before, (or) click Create Stack and select With new resource (standard)

• Under Prepare template, choose Template is ready

• Under Template source, choose Upload a template file

• Select Choose file

• Choose the downloaded .yaml template and then choose Open

• Choose Next

• For Stack name, enter a name for your template

• Set the following parameters:

  • AthenaCURBucket: The container into which the CUR is placed after the Step 1: CUR Setup
  • SpotDataFeedBucketName (optional): The bucket where the spot data feed is available. Refer Spot Data Feeds

• Choose Next and again Next

• Select I acknowledge that AWS CloudFormation might create IAM resources at the bottom of the page
• Choose Create Stack

Kubernetes clusters run in different accounts from the master payer account

Perform the below steps on each sub-account that runs Cost Management:

• Download this file
• Navigate to https://console.aws.amazon.com/cloudformation
• Choose Create New Stack if not used AWS CloudFormation before, (or) choose Create Stack
• Under Prepare template, choose Template is ready
• Under Template source, choose Upload a template file
• Select Choose file
• Choose the downloaded .yaml template, and then choose Open
• Choose Next
• For Stack name, enter a name for your template

• Set the following parameters:

  • MasterPayerAccountID: The account ID of the master payer account where the CUR has been created
  • SpotDataFeedBucketName: The bucket where the spot data feed is available. Refer Spot Data Feeds

• Choose Next and again Next

• Select I acknowledge that AWS CloudFormation might create IAM resources available at the bottom of the page
• Choose Create Stack

Perform the below steps on the master payer account

• Follow the same steps to create a CloudFormation stack as above, but with the yaml file instead along with the below parameters

  • AthenaCURBucket: The bucket where the CUR is set (Step 1: CUR Setup)
  • AWSAccountID: AWS account where cluster with Cost Management runs requires access to the Athena CUR

---

Add Manually

Users can also add manually to set the IAM roles.

Kubernetes clusters run in the same account as the master payer account

Perform the below steps:

• Attach the below two policies to the same role or user. Pick a user to integrate via servicekey (or) a role to integrate via IAM annotation. The SpotDataAccess policy statement is optional if the spot data feed is configured

Policy 1

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "AthenaAccess",
         "Effect": "Allow",
         "Action": [
            "athena:*"
         ],
         "Resource": [
            "*"
         ]
      },
      {
         "Sid": "ReadAccessToAthenaCurDataViaGlue",
         "Effect": "Allow",
         "Action": [
            "glue:GetDatabase*",
            "glue:GetTable*",
            "glue:GetPartition*",
            "glue:GetUserDefinedFunction",
            "glue:BatchGetPartition"
         ],
         "Resource": [
            "arn:aws:glue:*:*:catalog",
            "arn:aws:glue:*:*:database/athenacurcfn*",
            "arn:aws:glue:*:*:table/athenacurcfn*/*"
         ]
      },
      {
         "Sid": "AthenaQueryResultsOutput",
         "Effect": "Allow",
         "Action": [
            "s3:GetBucketLocation",
            "s3:GetObject",
            "s3:ListBucket",
            "s3:ListBucketMultipartUploads",
            "s3:ListMultipartUploadParts",
            "s3:AbortMultipartUpload",
            "s3:CreateBucket",
            "s3:PutObject"
         ],
         "Resource": [
            "arn:aws:s3:::aws-athena-query-results-*"
         ]
      },


      {
         "Sid": "S3ReadAccessToAwsBillingData",
         "Effect": "Allow",
         "Action": [
            "s3:Get*",
            "s3:List*"
         ],
         "Resource": [
            "arn:aws:s3:::${AthenaCURBucket}*"
         ]
      }
   ]
}

Policy 2

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "SpotDataAccess",
         "Effect": "Allow",
         "Action": [
            "s3:ListAllMyBuckets",
            "s3:ListBucket",
            "s3:HeadBucket",
            "s3:HeadObject",
            "s3:List*",
            "s3:Get*"
         ],
         "Resource": "arn:aws:s3:::${SpotDataFeedBucketName}*"
      }
   ]
}

Kubernetes clusters run in different accounts

Perform the following steps:

• On each sub account that runs Cost Management, attach the below 2 policies to the same role or user. Pick a user to integrate via servicekey (or) a role to integrate via IAM annotation. The SpotDataAccess policy statement is optional if the spot data feed is configured

Policy 1

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "AssumeRoleInMasterPayer",
         "Effect": "Allow",
         "Action": "sts:AssumeRole",
         "Resource": "arn:aws:iam::${MasterPayerAccountID}:role/costmanagementRole-${This-account’s-id}"
      }
   ]
}

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "SpotDataAccess",
         "Effect": "Allow",
         "Action": [
            "s3:ListAllMyBuckets",
            "s3:ListBucket",
            "s3:HeadBucket",
            "s3:HeadObject",
            "s3:List*",
            "s3:Get*"
         ],
         "Resource": "arn:aws:s3:::${SpotDataFeedBucketName}*"
      }
   ]
}

Policy 2

Attach this policy to a role on the masterpayer account (replace ${AthenaCURBucket} variable)

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "AthenaAccess",
         "Effect": "Allow",
         "Action": [
            "athena:*"
         ],
         "Resource": [
            "*"
         ]
},
{
         "Sid": "ReadAccessToAthenaCurDataViaGlue",
         "Effect": "Allow",
         "Action": [
            "glue:GetDatabase*",
            "glue:GetTable*",
            "glue:GetPartition*",
            "glue:GetUserDefinedFunction",
            "glue:BatchGetPartition"
         ],
         "Resource": [
            "arn:aws:glue:*:*:catalog",
            "arn:aws:glue:*:*:database/athenacurcfn*",
            "arn:aws:glue:*:*:table/athenacurcfn*/*"
         ]
      },
      {
         "Sid": "AthenaQueryResultsOutput",
         "Effect": "Allow",
         "Action": [
            "s3:GetBucketLocation",
            "s3:GetObject",
            "s3:ListBucket",
            "s3:ListBucketMultipartUploads",
            "s3:ListMultipartUploadParts",
            "s3:AbortMultipartUpload",
            "s3:CreateBucket",
            "s3:PutObject"
         ],
         "Resource": [
            "arn:aws:s3:::aws-athena-query-results-*"
         ]
      },
      {
         "Sid": "S3ReadAccessToAwsBillingData",
         "Effect": "Allow",
         "Action": [
            "s3:Get*",
            "s3:List*"
         ],
         "Resource": [
            "arn:aws:s3:::${AthenaCURBucket}*"
         ]
      }
   ]
}

Now, you must add the following trust statement to the role the policy is attached to (replace ${costmanagementClusterID} variable):

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": {
            "AWS": "arn:aws:iam::${AWSAccountID}:root"
         },
         "Action": [
            "sts:AssumeRole"
         ]
      }
   ]
}

---

Step 4: Attach IAM permissions to Cost Management

Once the policies are created, users can attach those policies to Cost Management. The following methods are supported:

Via Access Key

• Login to AWS IAM Access page
• Navigate to Access Management -> Users
• Find the Cost Management User and select Security Credentials -> Create Access Key. Note the Access key ID and Secret access key. You will use it to either Create a secret from helm values or Create and use an existing secret
• Attach this Access ID and Secret key, and create a Cost Management AWS Cloud Credential

Via Role ARN

• Create an OIDC provider for your cluster
• Create a Role
• Attach the policies created in Step 3
• Attach this Role ARN when creating a cloud credentials of type Cost Management and provider AWS

---

Step 5: Configure CUR config values to Cost Management

• Athena Bucket Name: An S3 bucket to store Athena query results to which Cost Management has permission to access

• The name of the bucket should match s3://aws-athena-query-results-*, so the IAM roles defined in Step 3 will automatically allow access to it

• The bucket can have a Canned ACL of Private or other permissions

• Athena Region: The AWS region Athena is running in

• Athena Database: The name of the database created by the Athena setup

• The Athena database name is available as the value (physical id) of AWSCURDatabase in the CloudFormation stack created in Step 2

• Athena Table: The name of the table created by the Athena setup

• The table name is typically the database name with the leading athenacurcfn_ removed (but is not available as a CloudFormation stack resource)

• Master Payer Account ARN: If using a multi-account setup, it is mandatory to set to the Amazon Resource Number (ARN) of the role in the masterpayer account. For example, arn:aws:iam::530337586275:role/costmanagementRole

---

Spot Data feed integration

Cost Management reconciles the users' spot prices with CUR billing reports as they become available (usually 1-2 days), but pricing data can be retrieved hourly by integrating directly with the AWS spot feed. To enable, follow the steps provided here

---