How to build and deploy Docker images?
When working on: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10234 we wanted to deploy a CronJob to k8s. The current state is:
- the source of the script sits in the runbooks repo
- the Dockerfile, the corresponding CI config and the Docker images are all in
ci-images
project - kubernetes config is in k8s-workloads
Selected issues with this approach:
- CI jobs for rebuilding the image are not automatically triggered by pushes to the runbooks repo (we probably could set up a job in the runbooks repo that would trigger a pipeline in the
ci-images
repo, but that's not very clean) - Even if you manually re-trigger the job to build a docker image, the SHA of the commit will not change, so the SHA tag will not change and as a result, the image in the registry will be overwritten (but the docker daemon on k8s nodes will still have the old version of the image and not pull the new image). This encourages pushing empty changes to master of the
ci-images
project whenever you want to rebuild an image. - CI jobs for deploying the image to k8s are not automatically triggered (you need to manually make a commit in a k8s repo changing the tag of the image)
- CI pipelines in runbooks are very slow so it takes a lot of time to make any changes
- one needs to work with three repos to deploy a change
Ideas:
- stick with a runbooks mono-repo approach (build and publish images in the runbooks repo), improve the CI config in the runbooks repo so that pipelines are faster, add automation for deployments
- start doing polyrepo (code+ci+registry for each image in an individual repo)
- start a mono-repo from scratch containing only images
- use the
gitlab-helmfiles
project for hosting code+ci+registry for all images and in the future include automation for deployment so that agit push
results in a full CI pipeline that includes: tests, Docker image building, deployment to staging, manual job for deploying to prod
@ggillies @jarv @skarbek @craigf @igorwwwwwwwwwwwwwwwwwwww for ideas and discussion (and possibly existing patterns in place that I'm not aware of)
Edited by Michal Wasilewski