Kubernetes and GitLab Docker Registry Access

Problem Statement

We run multiple Kubernetes Cluster (EKS based) in our production environment. Today we have stored all Docker images in the included GitLab Container Registry. The registry itself has stored all data in S3. In the beginning it was not possible to create new deploy tokens via API, so we decided to have a dedicated GitLab Bot User who has access to all projects in use.

For now we have two problems:

• since the Container Registry is not high-available, we have a massiv single-point of failure called GitLab [1]
• deploy tokens are GitLab project based or group based, since we have workloads with multiple containers per pod and we bound a ImagePullSecret to the default service account, with deploy tokens we have to move the imagePullSecret into the deployment definition (we don't like this)

What we want is the following:

• a Container Registry integrated into our GitLab workflows but highly available (EC2 instances in ASG, S3 backend)
• a Container Registry authentication module without dependency to the GitLab API [2]
• a single imagePullSecrets per k8s Namespace with access to a list of repositories

[1] We know, that we can move to GitLab Geo with the premium licence, but we a active/standby setup doesn't provide enough availability and reliability we need. [2] Our cluster are pretty dynamic in terms of capacity. We run node autoscaler and pod autoscaler to manage our load. We also run the descheduler. This means, we have hundreds of image pulls per day and all rely on the availability of GitLab.

Proposed Solution (we want to verify this with you)

We run multiple EC2 nodes in a ASG behind and ELB. Each node runs the Docker Container Registry (S3 storage backend) and a simple go http service (we call it CRA = container-registry-auth). Instead of https://<gitserver>/jwt/auth, we use our CRA here. The CRA has two authentication modes:

A) proxy mode to GitLab (to allow the standard GitLab authentication procedures) B) a JWT based authentication / authorisation mode

The JWT is issued by an internal provisioning tool (this tool is creating namespaces, managing AWS policies, etc) and it is signed by an RSA private key. It also included the the authorisation information (list of repositories). The CRA is validating (authentication) with the RSA public key the JWT and is authorising the docker pull request (match the request repository to one of the repositories in the JWT).

Benefits

• no direct dependency to GitLab during docker login and docker pull
• a single JWT token can authorise the access to multiple repositories

Problems we see

• there is a fork of the Docker Container Registry and we are not sure about the direction of this move and what implication this in the future might have
• we believe as long as we run in a read-only mode here, we will not run in any consistency issues (needs to be validated)
• in GitLab Geo there is the need for configuring "registry notification" - not sure why this is need and how this is usable with dynamic number of Docker Container Registries (ASG)

Thanks for feedback.

mentioned in issue gitlab-org/quality/triage-reports#508 (closed)

mentioned in issue gitlab-org/quality/triage-reports#510 (closed)

added Category:Container Registry devopspackage grouppackage registry typefeature labels

@nils.juenemann Thank you for opening up this thorough issue. I think this touches a few different teams here at GitLab, so I'll cc a few folks here for visibility.

there is a fork of the Docker Container Registry and we are not sure about the direction of this move and what implication this in the future might have

If you are interested, I'd be happy to jump on a call and talk this through with you.

• @jreporter for the Deploy Tokens feedback
• @fzimmer for the Geo feedback
• @joshlambert for HA questions
• @jdrpereira for some of your registry concerns.

thanks @trizzi for the tag

deploy tokens are GitLab project based or group based, since we have workloads with multiple containers per pod and we bound a ImagePullSecret to the default service account, with deploy tokens we have to move the imagePullSecret into the deployment definition (we don't like this)

This makes a lot of sense.

since the Container Registry is not high-available, we have a massiv single-point of failure called GitLab

Thanks. One can run multiple pods of the Registry which should provide some resilience for pod/node outages. We do this today on GitLab.com, and by default you get a minReplica of 2: https://docs.gitlab.com/charts/charts/registry/index.html#configuration

Similarly the other components of GitLab can be scaled up, and provided you have resilience in the database (say with RDS) you should have a fairly fault tolerant set up. If you are on VM's/EC2 for your GitLab server, you can similarly run multiple copies of the Registry/GitLab service to make it fault-tolerant.

in GitLab Geo there is the need for configuring "registry notification" - not sure why this is need and how this is usable with dynamic number of Docker Container Registries (ASG)

@nhxnguyen - do you know the answer to this one?

in GitLab Geo there is the need for configuring "registry notification" - not sure why this is need and how this is usable with dynamic number of Docker Container Registries (ASG)

I believe the notification events are needed for the secondary to sync any updates. But I'm not sure about whether it's usable with a dynamic number of registries. @mkozono or @dbalexandre do you know?

I believe the notification events are needed for the secondary to sync any updates.

Yes, the notification events are needed for the secondary node to sync any updates, e.g., whenever a new image is pushed to the primary node, each secondary node will pull it to its own container repository.

But I'm not sure about whether it's usable with a dynamic number of registries.

In the scenario described in this issue, where we have a dynamic number of Docker Container Registries, we can point to all of them to the same distributed storage (S3). @vsizov What do you think?

"registry notification" is only needed if you plan to use GitLab Geo. In other words, if your cluster of Registries will be connected to a Primary Geo node then every writable Regsitry should have configured section "notification" so all the pushes are replicated to secondary.

changed milestone to %Backlog

added [deprecated] Accepting merge requests label

added sectionops label

@nils.juenemann the registry can be integrated with any auth service that conforms to the spec, which I assume your CRA service does.

As Joshua mentioned above, you can achieve HA by having multiple registry instances behind a reverse proxy.

we believe as long as we run in a read-only mode here, we will not run in any consistency issues (needs to be validated)

Can you expand on this? I assume by read-only mode you are referring to the registry read-only mode, which is most commonly used to run offline garbage collection.

Closing due to inactivity.

closed