Determine replacement for helm-git

Currently the repos k8s-workloads/gitlab-com and k8s-workloads/gitlab-helmfiles both leverage the helm-git plugin to allow us to reference git repositories on dev.gitlab.org as sources for helm charts.

For this issue, I am going to focus less on gitlab-helmfiles as I think that repo is likely to be consumed by tanka-deployments and just focus on usage in gitlab-com.

The helm-git plugin currently has a few issues, namely incompatibility issues and bugs with newer versions of the plugin (0.10.0) and newer versions of helm (3.4.X and above). See

This is causing us a few issues

  • We can't update the plugin version of helm-git
  • We can't update the version of helm
  • We often get questions from people trying to onboard themselves to doing Kubernetes work for gitlab.com, and getting one of the errors above (because they install the latest version of the plugin, 0.10.0, instead of an old version that still works, 0.8.0). We have documented to use the old version of the plugin, but it is something that is easy to mix (because why shouldn't we be using the latest versions of plugins? Especially as we can't version them in .tool-versions)

Looking at the code for the plugin (bash), the main things it does (that we are interested in) is it

  • Clones a git repo of the chart, and makes sure it's at the specific commit we need
  • runs helm dependency build to download .tar.gz files for any missing dependent charts.

While I do have hope upstream will eventually fix the issues, they have been around for months now with no indication they will be fixed soon. Seeing as what the plugin does is relatively simple, and it's causing us maintainability pain, I would like to get a solution for this so we can move forward with blocked issues like helm upgrades.

Note: we currently have to do chart bumps across all environments in 1 MR, while we are tackling changing how we do chart bumps, we should also make sure the strategy supports chart bumps per environment. I believe all possible options can support this

Chosen solution

After evaluating the options (below) we will try the approach of vendoring with git subtree (option 5). Work is being tracked in #1959 (closed)

Evaluated solutions

  1. For the helm-git plugin and/or try and fix it ourselves

    • I am not entirely convinced that keeping a helm "plugin" that people have to install/upgrade (especially as we can't leverage things like asdf) for such simple functionality makes sense

  2. Switch to using git submodules

  3. Vendor the charts directly into the gitlab-com repo

    • This approach has a lot of benefits.
      • It's fast, means we don't have to burn time during k-ctl constantly re-downloading the chart (either in pipeline or doing local development)
      • We know a pipeline will never fail because of chart download, no dependency on Gitlab.com during emergencies, etc.
      • Tanka requires you to vendor charts inside the repo (and for very sound reasons), so we would be adopting the strategy also used for tanka-deployments
      • For people not familiar with the gitlab-com repo, this makes it much easier for them to understand what is going on. Every single file that goes into the "end result" is there inside the repo. It can be confusing to grep for something you see running in production and not find it, because the final manifests are kept in the helm chart in an entirely other repo
      • It's simple for EOCs, especially working in different environments (e.g. on a console server). You cone this repo and you know you have everything you need, you don't need to setup your ssh key to access dev.gitlab.org (as an example)
    • Some cons
      • It does make chart bump MRs a little more "ugly" (as you will have big diffs from the upstream chart changes), and the act of a chart bump goes from simply changing the git SHA to actually having to rm -rf chart/gitlab; git clone https://dev.gitlab.org/gitlab/charts/gitlab charts/gitlab;cd charts/gitlab && git checkout $SHA && cd ../../;
      • however we could write a helper in bin to make this 1 step (perhaps even creating the branch, doing the bump, and opening the MR in 1 tool!)
      • It's possible people could manually change something in the vendored copy of the chart, however this is easy to stop, we have a CI job that is dependant on changes: to charts/* that will simply compare the directory contents to a cloned copy (at the correct SHA) to confirm if you do change something in there, it still matches upstream (unless we want to allow deviations in case of emergency?)

  4. Add a helmfile pre-exec hook (or modify bin/get-chart to work as a hook) to check if a local copy of the chart exists (e.g. charts/gitlab) and if it doesn't, clone it, checkout the right SHA, and run helm dependency build.

    • Pros
      • Not vendoring the entire chart in the repo, but it's a similar result after you run helmfile at least once. The full chart will be there locally, just not committed to the repo (via .gitignore). This means only downloading the chart once (unless chart version changes) for most people, and CI can just leverage Caching to avoid having to download the chart over and ov er as well
      • Chart bumps look simple and clean (1 line change to a SHA)
    • Cons
      • Writing another bash hook we have to maintain, but it's arguable considering the current state of things if this is actually a con (not a pro)

  5. Utilise git subtree to vendor the chart in repo, but in a more Git native way (similar to git submodules)

    • Seems like just an all round better approach than git submodules.
    • Pros
      • Using git native tools, should minimise custom tooling and code we need to maintain
      • Actually stores the files in the main git repo (e.g. gitlab-com), so the pros of having a local copy already available when you checkout apply here (speed, ease of understanding, no external dependency in pipelines)
    • Cons
      • Diffs for chart bumps look like they still contain full changes from upstream
      • Would still need tooling to ensure people don't make local changes to the git subtree?
Edited by Amy Phillips