Deploying RC's for gitlab-shell is broken

GitLab Shell uses it's own tagging schema. When we recently created an RC, that RC is going to utilize the official version of GitLab Shell when building the Docker image. This can be seen in the associated build job: https://dev.gitlab.org/gitlab/charts/components/images/-/jobs/9546714

Note that on line 200, it was tagged 13.16.1, which matches the associated version that is published to the GitLab codebase: https://gitlab.com/gitlab-org/gitlab/-/blob/64a03133655f3a481c261b48f865060fbd1a8920/GITLAB_SHELL_VERSION

Prior to today, there was a bug that allowed us to skip deploying the correct version of GitLab Shell to the preprod environment. This bug attempted to be fixed by resuing the GitLab tag version. However, since GitLab Shell does not follow this same version, this led to a failure on the deployment.

The introduced "fix": gitlab-com/gl-infra/k8s-workloads/gitlab-com!697 (merged)

All future RC deploys will fail because we are not using the correct image tag for GitLab Shell.

Utilize this issue to determine how we can deploy the correct version of this component into preprod.

Notes

This only impacts preprod. The release instance does not run Kubernetes. For auto-deploys, we've forced CNG to build the docker images with a predefined auto-deploy tag that we use everywhere.

Consider asking CNG to build a tag for the RC version. This has the downside that the RC version of GitLab is BELOW the current version of GitLab Shell 😬

Consider trying to figure out of we can tell helmfile, how to find the appropriate version it ought to deploy.

Because we use an unreleased version of the helmchart, it wants to deploy the default of the chart which is master it appears: https://gitlab.com/gitlab-org/charts/gitlab/-/blob/0151a001f63ca00c09272aa2befc2c59d40fea6f/doc/charts/gitlab/gitlab-shell/index.md#installation-command-line-options

Workaround for 13-9-0-rc42

We created a docker tag that allowed us to get beyond this. This should NOT be done in the future

1m8s gitlab-com master● % docker pull dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-shell:v13.16.1
9m1s gitlab-com master● % docker tag dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-shell:v13.16.1 dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-shell:v13.9.0-rc42
12s gitlab-com master● % docker push dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-shell:v13.9.0-rc42

CC'ing a few people for potential insights cc @hphilipps @twk3 @WarheadsSE

Plan of Action/Milestones

Edited by Amy Phillips