Lots of users operate GitLab in restricted networks, especially government and financial services, where there is no Internet connection available. RedHat provides utilities and instructions for exporting operators from their Internet registries and importing them into a disconnected mirror registry.
In the code, the environment variables would evaluate to something like "amd64-v14.3.2" for the bundle image. OpenShift won't map that to the image on your disconnected mirror registry. OpenShift needs a digest (SHA) value.
For now I guess I'll have to stick with using a Helm chart (and adjusting it to pull from my mirror registry). This approach loses quite a lot; the operator would have been so much better...
Proposal
Update the GitLab Runner Operator to meet Red Hat's Air-Gap requirements.
The idea is that all the official images are pulled with a sha, but I see that the operator container image is not containerImage: 'registry.connect.redhat.com/gitlab/gitlab-runner-operator:v1.4.0'
I'll evaluate if there are others, this is not the intended behavior, thank you for reporting!
It doesn't look like there's any plan to get this resolved. Can we get it bumped up? With this issue, the OpenShift operator cannot be installed into OpenShift unless there's an active Internet connection. Closed networks are a big use case for government and financial sector.
@DarrenEastman is there a documented known workaround for the problem this issue addresses? This was recently postponed many months, which likely means it will get pushed again in a few months.
@brett_jacobson Good question on the documentation approach for a workaround. From my understanding of the requirements for air-gapped environments, probably not.
@ggeorgiev_gitlab wdyt- any known workarounds or is the only solution to fix the images?
Hey, can you tell me why the Operator wouldn't install in your air-gapped environments?
In the last published version of the official Operator https://github.com/redhat-openshift-ecosystem/certified-operators/pull/854/files, if you expand the file operators/gitlab-runner-operator/1.9.0/manifests/gitlab-runner-operator.clusterserviceversion.yaml and search for registry.gitlab.com, you'll see that all the images are using their respective sha to be pulled instead of a tag, so there must be some other reason, rather than the original report
I can confirm that our ClusterServiceVersion also has sha256 references (although different ones) when installing Gitlab Runner Operator 1.9.0 on OpenShift. However, as soon as you create a Runner, the runner pod that is created references registry.gitlab.com/gitlab-org/ci-cd/gitlab-runner-ubi-images/gitlab-runner-ocp:amd64-v15.0.0 ... and the helper image is also tagged amd64-v15.0.0
@ggeorgiev_gitlab thank you for these details!
Yes, we were tracking the Runner configs that allow for this, but we allow our tenants to create their own Runners and try to keep their definitions free of these kinds of cumbersome details.
In the meantime, we did track down the digests, and we use a Kyverno policy to automatically inject the correct @sha256 digests on the runner and helper, but again, we treat this as a band-aid that we would love to remove one day and not carry around this "hack"/technical debt.
@ggeorgiev_gitlab The GitLab UltimateCustomer who requested this is stating this is still an issue in Operator v1.10.0, stating "Creating a runner with the default image that is used has a tag rather than a digest, which is the same behavior experienced in v1.9.0"
I wonder, how are most people getting the operator into their disconnected environments? Are you building it straight from source (e.g. from the official repo at Red Hat's repository, as per the link above)?
Should we just consider the current HEAD commit as being our 1.10.0? Should we just use whatever catalog was most recently pushed?
It's not really straightforward to determine what catalog to use with labels like "amd64-v0.0.1-e76416e3." Mostly I'm just comparing dates vs. tags and making a buest guess.
Just to add some detail and clarity, when we tested v1.10.0, we mirrored the operator using Red Hat's oc-mirror tool/process as linked above. It mirrored the images properly, and I can install it from scratch, but I could not upgrade (had to uninstall) and the actual runner/helper images that get launched still use tags rather than digests... (which I know can be overridden in Runner manifest but we'd truly prefer to avoid)
When I later import the results into my disconnected registry, the containers get imported but at the very end, oc-mirror logs an error and fails to create the two yaml files we need (i.e. the catalog source or image content policy files), which I would then have to create manually.
I do have to set --continue-on-error on oc-mirror to ignore a handful of other errors, but it does always generate the ICSP and CatalogSource files that I need.
we use ImageContentSourcePolicy in Openshift to rewrite a external registry url to our own, which then acts as proxy-cache and connects to the external registry. ImageContentSourcePolicy works ONLY with digest, that is why this fix is so crucial. that is why i opened #62 (closed) in may 2021... we've meanwhile lost confident in the gitlab-runners in openshift project, as the problem is not understood and properly solved from its devs.... this appears also here #123 (closed) ... its still an open issue end of 2022. which is really disapointing.
Installed version v1.10.0 on an OpenShift 4 fully disconnected cluster this week and hit this issue still. We had to use the workaround that is described in this ticket.
apiVersion: apps.gitlab.com/v1beta2kind: Runnermetadata: name: devspec: gitlabUrl: https://gitlab.example.com token: gitlab-runner-secret # Name of the secret containing the Runner token runnerImage: registry.gitlab.com/gitlab-org/ci-cd/gitlab-runner-ubi-images/gitlab-runner-ocp@sha256:9e301e5077e23e3264e84eea0c0f0072eaa44d3cd98dd41e193e0cd13bb9e121 helperImage: registry.gitlab.com/gitlab-org/ci-cd/gitlab-runner-ubi-images/gitlab-runner-helper-ocp@sha256:09ea89b7eea7d33a901a1b85f3b4e6b1c9ca0f03fe1ffd42204a454da51a82ec
@ggeorgiev_gitlab it seems that we are still getting reports of errors with installation. Do you recommend (1) opening another issue or (2) documenting any known workarounds?
I have the same GitLab UltimateCustomer who is experiencing the same issue with the new Openshift GitLaber runner Operator v1.12, and has had to revert back to v1.11. Is this bug already known?