Pushing Helm charts to AWS ECR with Gitlab-CI

- 6 mins read

The AWS ECR (Elastic Container Registry) is a managed container registry providing storage and access mechanisms for your container images or other OCI related assets. Typically, if I’ll be dependent upon an image I will use a CI pipeline to mirror this into a Gitlab container registry. In some cases though it’s also useful to mirror assets into the ECR. For example, this can improve availability of assets to cloud resources (such as processes running on EC2 nodes) and minimise traffic across the tenancy.

Setting up required resources

The approach being demonstrated uses terraform. It’s possible to use the AWS cli directly (see appendix A) and also creates ECR repositories directly in a CI job. The process for mirroring assets into ECR is essentially identical to the process for mirroring into a Gitlab container registry. The main difference is the need to establish some AWS resources as a prerequisite:

  • an AWS account (that won’t be covered here)
  • a policy that allows that grants permissions to manipulate the ECR
  • a group to attach the policy to
  • a user that we’ll designate specifically for accessing the ECR via CI
  • plumbing this into gitlab

We can create a policy either through the console or cli. What’s important is that we provide access to the ECR. We can use the following policy document:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "ListImagesInRepository",
			"Effect": "Allow",
			"Action": [
				"ecr:ListImages"
			],
			"Resource": "*"
		},
		{
			"Sid": "GetAuthorizationToken",
			"Effect": "Allow",
			"Action": [
				"ecr:GetAuthorizationToken"
			],
			"Resource": "*"
		},
		{
			"Sid": "ManageRepositoryContents",
			"Effect": "Allow",
			"Action": [
				"ecr:CreateRepository",
				"ecr:BatchCheckLayerAvailability",
				"ecr:GetDownloadUrlForLayer",
				"ecr:GetRepositoryPolicy",
				"ecr:DescribeRepositories",
				"ecr:ListImages",
				"ecr:DescribeImages",
				"ecr:BatchGetImage",
				"ecr:InitiateLayerUpload",
				"ecr:UploadLayerPart",
				"ecr:CompleteLayerUpload",
				"ecr:PutImage"
			],
			"Resource": "*"
		}
	]
}

The setup via Terraform

resource "aws_iam_group" "ecrusers" {
  name = "ECRUsers"
}

resource "aws_iam_policy" "ecraccess" {
  name        = "ecr-access"
  description = "provides full access to ECR"
  policy = jsonencode(
    {
      "Version" : "2012-10-17",
      "Statement" : [
        {
          "Sid" : "ListImagesInRepository",
          "Effect" : "Allow",
          "Action" : [
            "ecr:ListImages"
          ],
          "Resource" : "*"
        },
        {
          "Sid" : "GetAuthorizationToken",
          "Effect" : "Allow",
          "Action" : [
            "ecr:GetAuthorizationToken"
          ],
          "Resource" : "*"
        },
        {
          "Sid" : "ManageRepositoryContents",
          "Effect" : "Allow",
          "Action" : [
            "ecr:CreateRepository",
            "ecr:BatchCheckLayerAvailability",
            "ecr:GetDownloadUrlForLayer",
            "ecr:GetRepositoryPolicy",
            "ecr:DescribeRepositories",
            "ecr:ListImages",
            "ecr:DescribeImages",
            "ecr:BatchGetImage",
            "ecr:InitiateLayerUpload",
            "ecr:UploadLayerPart",
            "ecr:CompleteLayerUpload",
            "ecr:PutImage"
          ],
          "Resource" : "*"
        }
      ]
    }
  )
}

resource "aws_iam_group_policy_attachment" "ecraccess" {
  group      = aws_iam_group.ecrusers.name
  policy_arn = aws_iam_policy.ecraccess.arn
}

# ECR user for use with CI
resource "aws_iam_user" "ecrciuser" {
  name          = "ECRCIUser"
  force_destroy = true
}

resource "aws_iam_user_group_membership" "ecrciuser" {
  user   = aws_iam_user.ecrciuser.name
  groups = [aws_iam_group.ecrusers.name]
}

We can also create the repositories and configure cross-region replication:

# ECR Repositories
resource "aws_ecr_repository" "gitlab" {
  name = "gitlab"
  image_tag_immutability = "IMMUTABLE"
}

# Replication Rules
data "aws_caller_identity" "this" {}

resource "aws_ecr_replication_configuration" "this" {
  replication_configuration {
    rule {
      dynamic "destination" {
        for_each = var.ecr_regions
        content {
          region      = destination.value
          registry_id = data.aws_caller_identity.this.account_id
        }
      }

      repository_filter {
        filter      = "gitlab"
        filter_type = "PREFIX_MATCH"
      }
    }
  }
}

Gitlab-CI integration

Now we have a user with permissions to manipulate the ECR and an access key for that user. Now in the repository that will actually be associated with the mirroring pipeline we’ll need to add some CI variables. One containing the access key ID and another containing the access key secret. The ‘mask variable’ option should be enabled for both of these so they’ll be masked in job traces. We’ll assume they’re called something like ‘AWS_ECR_ACCESS_KEY_ID’ and ‘AWS_ECR_ACCESS_KEY_SECRET’.

From here we only have to make the job that actually mirrors the charts. However, that will require a little bit of additional setup because we’ll require a platform for the job that provides the required tools:

  • aws-cli
  • helm

The simplest way to do this is to define our own using a Dockerfile. I’ll be using an image based on alpine-linux in the interests of minimising the size:

FROM alpine:3.19

RUN apk update && apk add aws-cli helm

ENTRYPOINT "/bin/sh"

We can stick this into some repository (generally people will have a group where they like to put their containers) and define some job to build and push this into the Gitlab container registry. Doing that won’t be covered here but it’s a very similar process to what we’re attempting.

Assuming we’ve pushed that image up somewhere accessible to the runner then we can proceed (in the following job I’ll just write this as helm-ecr-image). The job we write will need to accomplish a few things:

  • authenticating to the ECR
  • checking the repositories in the ECR and either:
    • creating one if it doesn’t exist (see above)
    • or failing to push if it doesn’t exist
  • pulling charts from upstream and then pushing them

The job will also need some additional information:

  • the region that the ECR is in (variable ECR_REGION eg ’eu-west-1’)
  • the ECR itself (variable ECR eg ‘.dkr.ecr..amazonaws.com’)
.mirror-ecr:
  stage: mirror
  when: manual
  allow_failure: true
  image: helm-ecr-image
  variables:
    ECR_REGION: <REGION>
    ECR: <ECR>
  before_script:
    # setup credentials
    - mkdir -p ~/.aws
    - |
      cat > ~/.aws/credentials <<END
      [default]
      aws_access_key_id = ${AWS_ECR_ACCESS_KEY_ID}
      aws_secret_access_key = ${AWS_ECR_ACCESS_KEY_SECRET}
      END      
    # authenticate to ECR
    - aws ecr get-login-password --region ${ECR_REGION} | helm registry login --username AWS --password-stdin ${ECR}
    # check for the repository
    - aws ecr describe-repositories --region ${ECR_REGION} --repository-names ${CHART_NAME} || { echo "Repository ${CHART_NAME} does not exist in ${ECR_REGION}, exiting." && exit 1; }
  script:
    - helm repo add $REPO_NAME $REPO_URL
    - |
      for CHART_VERSION in $CHART_VERSIONS
        do
          helm pull $REPO_NAME/$CHART_NAME --version $CHART_VERSION
          helm push $CHART_NAME-$CHART_VERSION.tgz oci://${ECR}/
        done      

mirror-gitlab-chart:
  extends: .mirror-ecr
  variables:
    REPO_NAME: gitlab
    REPO_URL: https://charts.gitlab.io/
    CHART_NAME: gitlab
    CHART_VERSIONS: >
      7.1.1
      7.1.2
      7.3.0
      7.3.5
      7.6.0
      7.7.0
      7.7.3
      7.8.1
      7.8.2
      7.9.1
      7.9.2

When this job is executed, you’ll see that all these versions of the gitlab chart are pushed up into the ECR for easy access. These chart pushes could also be configured as multiple jobs to run in parallel using Gitlab-CI parallel matrix.

Setup with cli

Create the policy:

aws iam create-policy --policy-name='ecr-access' --description='provides full access to ecr' --policy-document='{"Version": "2012-10-17","Statement": [{"Sid": "ListImagesInRepository","Effect": "Allow","Action": ["ecr:ListImages"],"Resource": "*"},{"Sid": "GetAuthorizationToken","Effect": "Allow","Action": ["ecr:GetAuthorizationToken"],"Resource": "*"},{"Sid": "ManageRepositoryContents","Effect": "Allow","Action": ["ecr:CreateRepository","ecr:BatchCheckLayerAvailability","ecr:GetDownloadUrlForLayer","ecr:GetRepositoryPolicy","ecr:DescribeRepositories","ecr:ListImages","ecr:DescribeImages","ecr:BatchGetImage","ecr:InitiateLayerUpload","ecr:UploadLayerPart","ecr:CompleteLayerUpload","ecr:PutImage"],"Resource": "*"}]}'

The output of this command will display the ARN of the policy, which we’ll need later on. We’ll also create a group for the ECR user(s) and attach the policy to it:

aws iam create-group --group-name ECRUsers
aws iam attach-group-policy --group-name ECRUsers --policy-arn <ARN>

Now we’re ready to actually create the user and add it to the group:

aws iam create-user --user-name ECRCIUser
aws iam add-user-to-group --group-name ECRUsers --user-name ECRCIUser

Note that this user won’t require access to the console but it will require an access key (which is how the pipeline shell authenticates to the ECR). Make sure you save the output of this command, this is the only time you’ll see this key:

aws iam create-access-key --user-name ECRCIUser

We also need to create the repositories, we can do this manually for any that don’t exist:

aws ecr describe-repositories --region <ECR_REGION> --repository-names <REPO_NAME> || aws ecr create-repository --region <ECR_REGION> --repository-name <REPO_NAME>

As an additional step, if CRR is desired then a replication rule needs to be added to the ECR:

aws ecr put-replication-configuration --region <ECR_REGION> --cli-input-json '{"replicationConfiguration":{"rules":[{"destinations":[{"region": "<SOME_OTHER_REGION>","registryId":"<ACCOUNT ID>"}],"repositoryFilters":[{"filter":"<REPO_NAME>","filterType": "PREFIX_MATCH"}]}]}}'