Use commit SHA as merge-train container image tag instead of floating tag

Problem description

Currently, the merge-train project uses a floating tag (${CI_REGISTRY_IMAGE}:latest) for the container image used in its CI jobs (.gitlab-ci.yml#L51).

This has the following effects:

  • Whenever the build:image job runs on the master branch, it pushes a new image to the container registry which is immediately used.
  • It is difficult to track changes in the container image from Git history, since any pipeline that runs the build:image job will push and overwrite the container image in container registry
  • Changes to the container image occur when changes are made to the files in bin directory. The previous container image is lost, and we cannot rollback, as happened during production#20585 (closed). This can result in unexpected failures during incidents that delay incident resolution.
  • Rolling back to a previously used container image is difficult, if not impossible, because every run of build:image overwrites the same container image in the container registry.

Proposal

  • When building a container image, use the $CI_COMMIT_SHORT_SHA so that each built image is uniquely named
  • In (.gitlab-ci.yml#L51) and .gitlab-ci.yml#L83, use the image tagged with the $CI_COMMIT_SHORT_SHA.

Effects this will have:

  • Changes to any of these files will result in the build:image job being executed, which will build a new image. However, this new image will not be automatically used.
  • To use a new image, (.gitlab-ci.yml#L51) and .gitlab-ci.yml#L83 will need to be updated to use the new image. This gives us a clear history of changes to the CI image, and makes it easy to rollback to the previous image if required.