Decouple `k8s-workloads/gitlab-com` `helmfile` execution from syncing from chef every time
Currently we leverage helmfile
exec
function to pull in a lot of settings from chef as helm values. e.g. https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/values-from-external-sources.yaml.gotmpl
While I feel this has served us well so far, it might be time to reconsider this approach, and consider other options. This is because
- The amount of times setting change in chef and need to be changed in Kubernetes as well is actually not that quite high. The most common change is adding new Gitaly nodes
- We are very much skirting the border of what is best practice with regards to "Gitops" which is, by definition means what you have committed to git is representative of what is running, with no exceptions or outside sources of configuration providing unpredictable results
- A failure to talk to chef (either because of transient errors or not enough permissions) stops your entire
helmfile
execution - We really do a lot of ugly stuff trying to get the values we need and convert to what we want due to the limited restrictions of Golang templates. Hard to read and debug
- When a value does change in chef, we currently have an awkward workflow where we try and recommend to do "empty commits" to run pipelines applying the change, to have some kind of record in this repository when a value changed
- It happens rarely but you do get a situation where you are trying to merge a change to this repo you inadvertently bring in chef changes during your change, if your pipeline simply runs at the wrong time
-
helmfile
executions take a bit longer due to the repeated calls to Gitlab to pull values from chef.
What I am proposing we do is consider externalising the pulling of values from chef into some external tool (exactly what to be determined), and have that tool pull the values we need from chef, and output a plain yaml or json file (perhaps values-from-external-sources.yaml
or values-from-chef.yaml
) and then have helmfile consume that file directly as a values file. This file would also be stored in git and be part of the full git history.
We would also have a job in our CI pipeline that would run the tool itself, compare the output to the contents of the MR, and fail if they are different (saying for people to run the tool and commit the changes as part of the MR). This would ensure that values from chef are being updated in the local copy we maintain. We could make this job optional depending on parameters passed to the invocation of the pipeline, so pipelines as part of auto deploy would never be blocked due to a chef change.
This issue will not address gitlab-secrets
and its workflow with chef, but any outcome from this issue (such as external tooling to sync) could be used if we wish to re-address gitlab-secrets
in the future.
Would appreciate any thoughts or feedback @jarv @skarbek @craigf