Pushing Helm charts to AWS ECR with Gitlab-CI

- 7 mins read

The AWS ECR (Elastic Container Registry) is a managed container registry providing storage and access mechanisms for your container images or other OCI related assets. Typically, if I’m going to depend on an image, I’ll use a CI pipeline to mirror it into a Gitlab Container Registry. In some cases though it’s also useful to mirror assets into the ECR. For example, this can improve availability of assets to cloud resources (such as processes running on EC2 nodes) and minimise bandwidth requirements for the Gitlab instance.

Setting up required resources

There are a few ways to accomplish getting images into the ECR but they all boil down to pretty much the same steps and, once the prerequisite AWS resources exist, the CI job will be very similar to one that would push images into the Gitlab registry. Our prerequisites are as follows:

  • an AWS account
  • an IAM user that we’ll authenticate as in the CI job
  • an IAM group that the user will belong too
  • an IAM policy that will allow access to AWS ECR
  • a container for the CI job that provides the AWS cli (and helm)

In addition to this we also need the actual ECR repositories to exist for our images. Additionally we might like to replicate images to ECR repositories in other regions to improve availability.

We could do all of this manually using a few cli commands and actually create the repositories in the CI job as and when we need them. There might be better flexibility this way since we’re only interacting with the ECR on a single layer but it requires more manual management of resources. Otherwise if we have a backend setup or are willing to manage a statefile, then a simple terraform configuration can also declare these same resources. I’ll show both approaches here although I would usually opt for the terraform because I like having as much in a VCS like git as possible although, really terraform is simply making API calls to the same endpoints.

policy

We can create a policy either through the console or cli. What’s important is that we provide access to the ECR. We can use the following policy document:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "ListImagesInRepository",
			"Effect": "Allow",
			"Action": [
				"ecr:ListImages"
			],
			"Resource": "*"
		},
		{
			"Sid": "GetAuthorizationToken",
			"Effect": "Allow",
			"Action": [
				"ecr:GetAuthorizationToken"
			],
			"Resource": "*"
		},
		{
			"Sid": "ManageRepositoryContents",
			"Effect": "Allow",
			"Action": [
				"ecr:CreateRepository",
				"ecr:BatchCheckLayerAvailability",
				"ecr:GetDownloadUrlForLayer",
				"ecr:GetRepositoryPolicy",
				"ecr:DescribeRepositories",
				"ecr:ListImages",
				"ecr:DescribeImages",
				"ecr:BatchGetImage",
				"ecr:InitiateLayerUpload",
				"ecr:UploadLayerPart",
				"ecr:CompleteLayerUpload",
				"ecr:PutImage"
			],
			"Resource": "*"
		}
	]
}

Setup with cli directly

Firstly create the policy via cli:

aws iam create-policy --policy-name='ecr-access' --description='provides full access to ecr' --policy-document='{"Version": "2012-10-17","Statement": [{"Sid": "ListImagesInRepository","Effect": "Allow","Action": ["ecr:ListImages"],"Resource": "*"},{"Sid": "GetAuthorizationToken","Effect": "Allow","Action": ["ecr:GetAuthorizationToken"],"Resource": "*"},{"Sid": "ManageRepositoryContents","Effect": "Allow","Action": ["ecr:CreateRepository","ecr:BatchCheckLayerAvailability","ecr:GetDownloadUrlForLayer","ecr:GetRepositoryPolicy","ecr:DescribeRepositories","ecr:ListImages","ecr:DescribeImages","ecr:BatchGetImage","ecr:InitiateLayerUpload","ecr:UploadLayerPart","ecr:CompleteLayerUpload","ecr:PutImage"],"Resource": "*"}]}'

The output of this command will display the ARN of the policy, which we’ll need later on. We’ll also create a group for the ECR user(s) and attach the policy to it:

aws iam create-group --group-name ECRUsers
aws iam attach-group-policy --group-name ECRUsers --policy-arn <ARN>

Now we’re ready to actually create the user and add it to the group:

aws iam create-user --user-name ECRCIUser
aws iam add-user-to-group --group-name ECRUsers --user-name ECRCIUser

Note that this user won’t require access to the console but it will require an access key (which is how the pipeline will authenticate to the ECR). Make sure you save the output of this command, this is the only time you’ll see this key:

aws iam create-access-key --user-name ECRCIUser

We also need to create the repositories, we can do this manually for any that don’t exist:

aws ecr describe-repositories --region <ECR_REGION> --repository-names <REPO_NAME> || aws ecr create-repository --region <ECR_REGION> --repository-name <REPO_NAME>

As an additional step, if CRR is desired then a replication rule needs to be added to the ECR. This can be done via the console or cli.

aws ecr put-replication-configuration --region <ECR_REGION> --cli-input-json '{"replicationConfiguration":{"rules":[{"destinations":[{"region": "<SOME_OTHER_REGION>","registryId":"<ACCOUNT ID>"}],"repositoryFilters":[{"filter":"<REPO_NAME>","filterType": "PREFIX_MATCH"}]}]}}'

Setup with Terraform

The following configuration will accomplish all the same things as the above section except creating an access key for the ECRCIUser.

resource "aws_iam_group" "ecrusers" {
  name = "ECRUsers"
}

resource "aws_iam_policy" "ecraccess" {
  name        = "ecr-access"
  description = "provides full access to ECR"
  policy = jsonencode(
    {
      "Version" : "2012-10-17",
      "Statement" : [
        {
          "Sid" : "ListImagesInRepository",
          "Effect" : "Allow",
          "Action" : [
            "ecr:ListImages"
          ],
          "Resource" : "*"
        },
        {
          "Sid" : "GetAuthorizationToken",
          "Effect" : "Allow",
          "Action" : [
            "ecr:GetAuthorizationToken"
          ],
          "Resource" : "*"
        },
        {
          "Sid" : "ManageRepositoryContents",
          "Effect" : "Allow",
          "Action" : [
            "ecr:CreateRepository",
            "ecr:BatchCheckLayerAvailability",
            "ecr:GetDownloadUrlForLayer",
            "ecr:GetRepositoryPolicy",
            "ecr:DescribeRepositories",
            "ecr:ListImages",
            "ecr:DescribeImages",
            "ecr:BatchGetImage",
            "ecr:InitiateLayerUpload",
            "ecr:UploadLayerPart",
            "ecr:CompleteLayerUpload",
            "ecr:PutImage"
          ],
          "Resource" : "*"
        }
      ]
    }
  )
}

resource "aws_iam_group_policy_attachment" "ecraccess" {
  group      = aws_iam_group.ecrusers.name
  policy_arn = aws_iam_policy.ecraccess.arn
}

# ECR user for use with CI
resource "aws_iam_user" "ecrciuser" {
  name          = "ECRCIUser"
  force_destroy = true
}

resource "aws_iam_user_group_membership" "ecrciuser" {
  user   = aws_iam_user.ecrciuser.name
  groups = [aws_iam_group.ecrusers.name]
}

We can also create the repositories here and configure cross-region replication: note that for the following we’d provide a list of regions to replicate to eg ["eu-west-2","eu-west-1"].

# ECR Repositories
resource "aws_ecr_repository" "gitlab" {
  name = "gitlab"
  image_tag_mutability = "IMMUTABLE"
}

# Replication Rules
data "aws_caller_identity" "this" {}

resource "aws_ecr_replication_configuration" "this" {
  replication_configuration {
    rule {
      dynamic "destination" {
        for_each = var.ecr_regions
        content {
          region      = destination.value
          registry_id = data.aws_caller_identity.this.account_id
        }
      }

      repository_filter {
        filter      = "gitlab"
        filter_type = "PREFIX_MATCH"
      }
    }
  }
}

In the above, we’re creating a repository for gitlab helm charts, and replicating that repository to other regions (defined by var.ecr_regions).

Gitlab-CI integration

Now we have a user with permissions to manipulate the ECR and an access key for that user. Now in the repository that will actually be associated with the mirroring pipeline we’ll need to add some CI variables. One containing the access key ID and another containing the access key secret. The ‘mask variable’ option should be enabled for both of these so they’ll be masked in job traces. We’ll assume they’re called something like ‘AWS_ECR_ACCESS_KEY_ID’ and ‘AWS_ECR_ACCESS_KEY_SECRET’.

From here we only have to make the job that actually mirrors the charts. However, that will require a little bit of additional setup because we’ll require a platform for the job that provides the required tools:

  • aws-cli
  • helm

The simplest way to do this is to define our own using a Dockerfile. I’ll be using an image based on alpine-linux in the interests of minimising the size:

FROM alpine:3.19

RUN apk update && apk add aws-cli helm

ENTRYPOINT "/bin/sh"

We can stick this into some repository (generally people will have a group where they like to put their containers) and define some job to build and push this into the Gitlab container registry. Doing that won’t be covered here but it’s a very similar process to what we’re attempting.

Assuming we’ve pushed that image up somewhere accessible to the runner then we can proceed (in the following job I’ll just write this as helm-ecr-image). The job we write will need to accomplish a few things:

  • authenticating to the ECR
  • checking the repositories in the ECR and either:
    • creating one if it doesn’t exist (see above)
    • or failing to push if it doesn’t exist
  • pulling charts from upstream and then pushing them

The job will also need some additional information:

  • the region that the ECR is in (variable ECR_REGION eg ’eu-west-1’)
  • the ECR itself (variable ECR eg ‘.dkr.ecr..amazonaws.com’)
.mirror-ecr:
  stage: mirror
  when: manual
  allow_failure: true
  image: helm-ecr-image
  variables:
    ECR_REGION: <REGION>
    ECR: <ECR>
  before_script:
    # setup credentials
    - mkdir -p ~/.aws
    - |
      cat > ~/.aws/credentials <<END
      [default]
      aws_access_key_id = ${AWS_ECR_ACCESS_KEY_ID}
      aws_secret_access_key = ${AWS_ECR_ACCESS_KEY_SECRET}
      END      
    # authenticate to ECR
    - aws ecr get-login-password --region ${ECR_REGION} | helm registry login --username AWS --password-stdin ${ECR}
    # check for the repository
    - aws ecr describe-repositories --region ${ECR_REGION} --repository-names ${CHART_NAME} || { echo "Repository ${CHART_NAME} does not exist in ${ECR_REGION}, exiting." && exit 1; }
  script:
    - helm repo add $REPO_NAME $REPO_URL
    - |
      for CHART_VERSION in $CHART_VERSIONS
        do
          helm pull $REPO_NAME/$CHART_NAME --version $CHART_VERSION
          helm push $CHART_NAME-$CHART_VERSION.tgz oci://${ECR}/
        done      

mirror-gitlab-chart:
  extends: .mirror-ecr
  variables:
    REPO_NAME: gitlab
    REPO_URL: https://charts.gitlab.io/
    CHART_NAME: gitlab
    CHART_VERSIONS: >
      7.1.1
      7.1.2
      7.3.0
      7.3.5
      7.6.0
      7.7.0
      7.7.3
      7.8.1
      7.8.2
      7.9.1
      7.9.2

When this job is executed, you’ll see that all these versions of the gitlab chart are pushed up into the ECR for easy access. These chart pushes could also be configured as multiple jobs to run in parallel using Gitlab-CI parallel matrix.