CI: Stash CNG image SHA2 at start of pipeline
Summary
While the CI pipelines of this project face several challenges, a few of them could be better addressed with minor changes.
One of the most annoying to diagnose ones is "why is this not up", which often comes down to images shifting "under" the tags, and the fact that we have been setting a floating tag alongside image.pullPolicy: Always
.
I propose that we have a job at the start of the pipeline that will collect the SHA2 digests for all images the chart will be consuming, and then set those properties upon deployment. While doing this will not prevent every possible instance, it will greatly reduce the likelihood.
- This will ensure consistency of all images across any number of re-runs of pipeline Jobs, and across all environments.
- This job itself could be re-run if needed, to re-stash these values.
- Ideally, we prevent mis-aligned database migrations preventing service pods coming up, and causing pipeline problems upon retry.
- Setting the image shas will result in re-deploy of all services when their images change, resulting in consistency across all components.
Functional
-
Implement a CI Job that will run before deployments, and collect image digests into an artifact (possibly, a direct values file) - This should be relatively simple to collect, with
helm template
, fetching the images metadata withskopeo
, and then stashing into appropriate content. - We know where each image is defined in values, and where they are templated into the YAML output.
- This could be in bash / yq, or in Ruby via YAML and JSON gems.
- This should be relatively simple to collect, with
-
The deployment Jobs consume this output into their helm upgrade --install
-
We remove global.image.pullPolicy=Always
from CI
Edited by Mitchell Nielsen