Installation

Here are the detailed instructions for installation of Air Gapped Controller with a Custom ECR registry, in EKS environments.

Before Installation¶

The Prerequisites must be completed before installing the Air Gapped Controller.

Installation Steps¶

The following uses Terraform to initialize the infrastructure. These steps can be skipped if the infrastructure is already created.

Clone the Controller Terraform Package¶

Clone the Controller Terraform package from the following repository to the system running the Terraform commands.

git clone -b custom-registry-v1.23 https://github.com/RafaySystems/onprem-dev.git

Navigate to the EKS directory for the Terraform scripts.

cd onprem-dev/terraform/EKS

Customize Terraform TFVARS File¶

Copy and customize the terraform.tfvars file.

cp -rp terraform.tfvars-tmpl terraform.tfvars
vi terraform.tfvars

In the terraform.tfvars file, update the run_only_infra variable to true. This will bring up the infrastructure only.

Refer to Amazon EKS TFVARS for more information about the terraform.tfvars configuration.

## Set to true for only bringing up infrastructure
run_only_infra = true

Install Infrastructure¶

Initialize the Terraform code and download the required providers and versions.

terraform init

Run a Terraform plan to check the resources that will be created.

terraform plan

Create the infrastructure to bring up the Controller.

terraform apply -auto-approve

Image Sync Prerequisites¶

Create IRSA Role for Image Pull¶

Create an IRSA role for authenticating the user to push the images to the ECR registry. You can either use a script or manually create the role.

Using a Script¶

Refer to Amazon EKS IRSA Role for the script to automatically create the IRSA role using the EKS cluster name and the region.

Manual Creation¶

To manually create a policy, use the following example.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListImagesInRepository",
      "Effect": "Allow",
      "Action": [
        "ecr:ListImages",
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer"
      ],
      "Resource": "arn:aws:ecr:us-east-1:01234567890:repository/rafay/*"
    },
    {
      "Sid": "GetAuthorizationToken",
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken"
      ],
      "Resource": "*"
    }
  ]
}

Create a Role with the name rafay-ecr-irsa-role, attach the above policy, and trust the OIDC provider of the EKS controller, see the example below.

Note

Make sure the name of the role is set to rafay-ecr-irsa-role, as this is referred to internally to pull the images.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::01234567890:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/11C92200000000000000000005A52"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-west-1.amazonaws.com/id/11C92200000000000000000005A52:sub": [
            "system:serviceaccount:rafay-registry:nexus-registry",
            "system:serviceaccount:rafay-registry:sa-health-check"
          ],
          "oidc.eks.us-west-1.amazonaws.com/id/11C92200000000000000000005A52:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

IAM Instance Role Setup in Local Instance¶

The Controller supports using an IAM role to push images to ECR. Create an IAM role with the following permissions and assign it to the local instance where the RADM image sync is done.

While running the radm registry sync command, --instance-role needs to be passed.
Role should have the following policy attached to the EC2 instance.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RafayECRImagesPolicy",
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetRepositoryPolicy",
        "ecr:DescribeRepositories",
        "ecr:ListImages",
        "ecr:DescribeImages",
        "ecr:BatchGetImage",
        "ecr:GetLifecyclePolicy",
        "ecr:GetLifecyclePolicyPreview",
        "ecr:ListTagsForResource",
        "ecr:DescribeImageScanFindings",
        "ecr:CreateRepository",
        "ecr:SetRepositoryPolicy"
      ],
      "Resource": "*"
    }
  ]
}

IAM Policy for ECR¶

Create an ecr-policy.json file with the following permissions for ECR.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "iam-user",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
             "arn:aws:iam::012345678900:role/rafay-ecr-push-policy",
             "arn:aws:iam::012345678900:user/[email protected]"
           ]
      },
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchDeleteImage",
        "ecr:BatchGetImage",
        "ecr:CompleteLayerUpload",
        "ecr:DeleteLifecyclePolicy",
        "ecr:DeleteRepository",
        "ecr:DeleteRepositoryPolicy",
        "ecr:DescribeImages",
        "ecr:DescribeRepositories",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetLifecyclePolicy",
        "ecr:GetLifecyclePolicyPreview",
        "ecr:GetRepositoryPolicy",
        "ecr:InitiateLayerUpload",
        "ecr:ListImages",
        "ecr:PutImage",
        "ecr:PutLifecyclePolicy",
        "ecr:SetRepositoryPolicy",
        "ecr:StartLifecyclePolicyPreview",
        "ecr:UploadLayerPart"
      ]
    },
    {
      "Sid": "iam-user",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
             "arn:aws:iam::012345678900:role/rafay-ecr-irsa-role",
             "arn:aws:iam::012345678900:user/[email protected]"
           ]
      },
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:DescribeImages",
        "ecr:DescribeRepositories",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetLifecyclePolicy",
        "ecr:GetLifecyclePolicyPreview",
        "ecr:GetRepositoryPolicy",
        "ecr:ListImages"
      ]
    }
  ]
}

Pushing Images to Custom Registry¶

Navigate to the directory where the Controller package was installed and execute the following commands to push the Controller and cluster images to the custom registry.

Registry Sync Controller Image to Custom Registry¶

./radm registry sync ecr --endpoint 012345678900.dkr.ecr.us-west-2.amazonaws.com --archive rafay-images.tar.gz --ecr-policy ecr-policy.json --ecr-region us-west-2 --stage /tmp --instance-role

Flags	Description
–-endpoint	The ECR endpoint (Required)
–-archive	The tarball shared by the Rafay containing the images (Required)
–-ecr-policy	The policy to be applied to the repo created for Rafay images (Required)
–-stage	The temporary directory to extract tarball on the local (default /tmp)
–-instance-role	To use ec2 instance role to run instead of AWS key and secret.
–-ecr-region	ECR region (Default us-west-2)

Registry Sync Cluster Image to Custom Repository¶

./radm registry sync ecr --endpoint 012345678900.dkr.ecr.us-west-2.amazonaws.com --archive rafay-cluster-images.tar.gz --ecr-policy ecr-policy.json --stage /tmp

Initialize Registry¶

Add IRSA Role to Config YAML¶

If using IAM roles to pull images, edit the generated config.yaml file and add the IRSA role ARN created in Create IRSA Role for Image Pull.

Note

If Terraform is used, the config.yaml, kube-config, and RADM files must be in the same directory as the terraform.tfvars file.

    rafay-registry:
      type: "ecr"
      registry-archive:
        path: /home/rafay/rafay-registry.tar.gz
        dir: /tmp
      ecr:
        aws-access-key: ""
        aws-secret-key: ""
        aws-irsa-role: "arn:aws:iam::01234567890:role/rafay-ecr-irsa-role"
        aws-region: "us-east-1"
        aws-ecr-endpoint: "01234567890.dkr.ecr.us-east-1.amazonaws.com"

Install Registry¶

Use the following command to install the registry. Change <kube-config-file> with the name of the kubeconfig file.

./radm registry configure --config config.yaml --kubeconfig ./<kube-config-file>

Update DNS to Registry Endpoint¶

Update the core-registry.<Controller-FQDN> A record to the loadbalancer created for the Nexus service in the rafay-registry namespace.

kubectl get svc -n rafay-registry nexus-service --no-headers | awk '{print $4}'
abcdefg0123456789012345678900-012345678900abc.elb.us-west-2.amazonaws.com

Verify that the Nexus pod is running.

kubectl get po -n rafay-registry
NAME                                 READY   STATUS      RESTARTS   AGE
nexus-0                              3/3     Running     0          15h

Install Controller¶

Make the following changes in the terraform.tfvars file to proceed with the Controller installation.

Update the terraform.tfvars with the IRSA role ARN created in Create IRSA Role for Image Pull.

##Update with irsa role arn created
ecr_aws_irsa_role = ""

Enable the application installation by switching the run_only_infra variable as false in the terraform.tfvars file.

## Set to true for only bringing up infrastructure
run_only_infra = false

Execute the Terraform apply command to bring up the Controller.

terraform apply

Update DNS Entries for Controller¶

Update the ui.<Controller-FQDN> A record to the IP address of the load balancer created for the istio-ingresssgateway service.

Get the load balancer details using the below command.

kubectl get svc -n istio-system istio-ingressgateway --no-headers | awk '{print $4}'

Update the backend.<Controller-FQDN> A record to the IP address of the load balancer created for the istio-ingresssgateway service. Use the following command.

kubectl get svc -n istio-system istio-ingressgateway-https --no-headers | awk '{print $4}'

Access Console¶

Access the console to verify a successful installation.

https://console.<Controller-FQDN>

The login screen displays.

Controller Login Screen

Click the Sign Up link to create the first organization and administrator for the Controller.

Controller Create Account

After creating an account, go back to the login page and use your credentials to log in.