Request to continue publishing helper images to Docker Hub
Status update (2022-12-29)
- We plan to restart the publishing of the helper image to Docker Hub
Description
Request to continue publishing gitlab-runner semantic version only helper images to Docker Hub. Publishing the helper images was halted but impacts a customer workflow that relies on disaster recovery and compliance scanning for their auditors.
Proposal
Continue publishing to Docker Hub until !3470 (closed) can be merged so that the defaults can be provided without modifying the config.toml
for every runner.
Links to related issues and merge requests / references
Affects Large Premium customer.
ZD (internal): https://gitlab.zendesk.com/agent/tickets/325772
SFDC (internal): https://gitlab.my.salesforce.com/00161000003QEedAAG
Customer impact
Removing the published image from Docker Hub does not impact the default configuration of GitLab runner, since that default does not use Docker Hub anymore, it does impact other workflows that rely on Docker Hub. In making this change, the runner team has broken the following:
- Our recovery storage of GitLab Runner Helper images that collects those images from Docker Hub and protects us from outages.
- Our compliance scanning of the images that uses the same mechanism.
- Our automation for determining the correct current version of images to use since GitLab Container Registry is not a compliant implementation of the Docker Registry API.
- Our ability to manually checkout GitLab Runner Helper images since GitLab Container Registry has a broken interface that does not search correctly or scale to the immense number of builds that are stored in it.
Customer workflow
- GitLab server instance hosted on-premise.
- There is a team responsible for the regular patching of the servers.
- Our process is to have the Runner version match the version of the GitLab instance.
- We don’t necessarily take on the latest release of GitLab Runner, so we are not able to rely on the latest image of Runner. Therefore, we need to find the gitlab-runner images that match the instance version. If done manually, the process falls apart.
- So our automation queries the DockerHub registry API and retrieve the list of images and compares the image versions.
- Then, we pull the Runner images from Docker Hub through Artifactory as a proxy. Artifactory is used for disaster recovery, so if DockerHub or GitLab runner container registry is not available, then we have images in Artifactory for our workflows.
What about changing the workflow to use the GitLab Container Registry?
Both parts of the automation do not work with GitLab container Registry.
- Proxy on Artifactory for Docker Hub.
- The code that retrieves the tags and versions does not work with the GitLab Container Registry.
- Additionally, we can’t use the same tagging as GitLab runner uses a different tag set
When did your automation break?
- This is another element of my frustration in that I did not know beforehand of the change and the potential impact to our workflows.
- When we went to 15.1 of Runner (July 2022) then, the automation broke.