push-helm-artifacts: add ability to disambiguate when two assets exist for the same (chart,version), use it for rke2-multus

Closes #4989 (closed)

Please see in #4989 (closed) the details about the context and why this being added.

This MR does the following:

  • evolve push-helm-chart-artifacts.py to let it pick a specific Helm chart archive when the Helm repo index has more than one for a given chart:version
    • the tool now accepts an optional DEDUP_HELM_CHART_IGNORE_URL, when set the index items that point to an archive matching this regex will be ignore
    • this is used only in the presence of duplicates
    • the "download archive" step in such a case will use a plain Python urllib.request instead of calling helm pull
  • in the absence of duplicate items, nothing changes, helm pull is still used
  • another addition was made to the tool to make it easy to test: the HELM_CHART_FILTER environment variable can be set to process only a subet of Helm charts (see example below)
  • to address the specific problem we have with rke2-multus-1.34-1.36, update GitLab CI job to use the new feature

Testing

This MR was tested with something reflecting the problem with rke2-multus Helm chart which lead to #4989 (closed) :

$ DEDUP_HELM_CHART_IGNORE_URL='1.34-1.36' \
  PUSH_HELM_ARTIFACTS_MODE=verify \
  HELM_CHART_FILTER='multus$' \
    tools/oci/push-helm-charts-artifacts.py

2026-11-13 11:52:05,259 [INFO] (working in /tmp/user/1000/sylva-artifact-5d0wk8f_)
--- processing unit multus
2026-11-13 11:52:05,819 [INFO] - processing version v4.3.102 from sylva-units values
2026-11-13 11:52:05,819 [INFO]   fetching index.yaml from https://rke2-charts.rancher.io/index.yaml
2026-11-13 11:52:07,735 [INFO]   index.yaml has 2 entries for rke2-multus:v4.3.102 - deduplicating with ignore-url regex '1.34-1.36'
2026-11-13 11:52:07,735 [INFO]     ignoring entry with urls: ['assets/rke2-multus-1.34-1.36/rke2-multus-v4.3.102.tgz']
2026-11-13 11:52:07,735 [INFO]   downloading https://rke2-charts.rancher.io/assets/rke2-multus/rke2-multus-v4.3.102.tgz -> /tmp/user/1000/sylva-artifact-5d0wk8f_/rke2-multus-v4.3.102.tgz
2026-11-13 11:52:07,991 [INFO] Checking if OCI artifact exists: rke2-multus:v4.3.102 :: oci://registry.gitlab.com/sylva-projects/sylva-core/rke2-multus
2026-11-13 11:52:07,991 [INFO]   running: helm pull oci://registry.gitlab.com/sylva-projects/sylva-core/rke2-multus --version v4.3.102 -d /tmp/user/1000/sylva-pulled-jloygn4e
2026-11-13 11:52:09,816 [WARNING]     !stderr: Pulled: registry.gitlab.com/sylva-projects/sylva-core/rke2-multus:v4.3.102
2026-11-13 11:52:09,816 [WARNING]     !stderr: Digest: sha256:256c4bf0f6c11b4fdd80c8e1e9b23e7ee15eeb913cfb565b65a71170300eb8e2
2026-11-13 11:52:09,817 [INFO] Checking the integrity of the existing artifact rke2-multus:v4.3.102 :: oci://registry.gitlab.com/sylva-projects/sylva-core/rke2-multus
2026-11-13 11:52:09,817 [INFO]   running: tar -xzf /tmp/user/1000/sylva-artifact-5d0wk8f_/rke2-multus-v4.3.102.tgz -C /tmp/user/1000/tgz-eq9oymyg
2026-11-13 11:52:09,822 [INFO]   running: tar -xzf /tmp/user/1000/sylva-pulled-jloygn4e/rke2-multus-v4.3.102.tgz -C /tmp/user/1000/sylva-pulled-jloygn4e
2026-11-13 11:52:09,826 [INFO] ---------- make a diff --------------
2026-11-13 11:52:09,826 [INFO]   running: diff -wur --exclude Chart.lock --exclude .git /tmp/user/1000/tgz-eq9oymyg/ /tmp/user/1000/sylva-pulled-jloygn4e/
2026-11-13 11:52:09,828 [INFO] Integrity check: ok

This new bit in logs is showing that this MR does what we want:

index.yaml has 2 entries for rke2-multus:v4.3.102 - deduplicating with ignore-url regex '1.34-1.36'
  ignoring entry with urls: ['assets/rke2-multus-1.34-1.36/rke2-multus-v4.3.102.tgz']

Development

Development was assisted by OpenCode/Claude Sonnet 4.6.

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon Meaning Available values
☁️ Infra Provider capd, capo, capm3
🚀 Bootstrap Provider kubeadm (alias kadm), rke2, okd, ck8s
🐧 Node OS ubuntu, suse, na, leapmicro
🛠️ Deployment Options Deployment option list and description
🎬 Pipeline Scenarios Available scenario list and description
🟢 Enabled units Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type
🔴 Disabled units Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type
🏗️ Target platform Can be used to select specific deployment environment Available platform list and description
🔀 Workload deployment workflow legacy (default; direct workload-cluster apply) or gitops (deploy workload cluster through the GitOps-managed flow)
⚡ Pipeline control autorun, manual or blocking. Can be used to override global config and start a deployment pipeline the required way
  • 🎬 preview ☁️ capd 🚀 kadm 🐧 ubuntu

  • 🎬 preview ☁️ capo 🚀 rke2 🐧 suse

  • 🎬 preview ☁️ capm3 🚀 rke2 🐧 ubuntu

  • ☁️ capd 🚀 kadm 🛠️ light-deploy 🐧 ubuntu

  • ☁️ capd 🚀 rke2 🛠️ light-deploy 🐧 suse

  • ☁️ capo 🚀 rke2 🐧 suse

  • ☁️ capo 🚀 rke2 🐧 leapmicro

  • ☁️ capo 🚀 kadm 🐧 ubuntu

  • ☁️ capo 🚀 kadm 🐧 ubuntu 🔀 gitops

  • ☁️ capo 🚀 kadm 🐧 ubuntu 🟢 neuvector,mgmt:harbor

  • ☁️ capo 🚀 rke2 🎬 rolling-update 🛠️ ha 🐧 ubuntu

  • ☁️ capo 🚀 kadm 🎬 wkld-k8s-upgrade 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🎬 rolling-update-no-wkld 🛠️ ha 🐧 suse

  • ☁️ capo 🚀 rke2 🎬 sylva-upgrade 🛠️ ha 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🎬 sylva-upgrade-from-1.6.x 🛠️ ha,misc 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🛠️ ha,misc 🐧 ubuntu

  • ☁️ capo 🚀 rke2|okd 🎬 no-update 🐧 ubuntu|na

  • ☁️ capo 🚀 rke2 🛠️ misc 🐧 ubuntu 🟢 mgmt:harbor 🔴 neuvector

  • ☁️ capo 🚀 rke2 🛠️ ha,misc,openbao 🐧 suse

  • ☁️ capo 🚀 rke2 🐧 suse 🎬 upgrade-from-prev-tag

  • ☁️ capm3 🚀 rke2 🐧 suse

  • ☁️ capm3 🚀 kadm 🐧 ubuntu

  • ☁️ capm3 🚀 rke2 🛠️ external-openbao-no-tls 🐧 ubuntu

  • ☁️ capm3 🚀 rke2 🛠️ external-openbao 🐧 suse

  • ☁️ capm3 🚀 kadm 🛠️ external-openbao,openbao-namespace 🐧 ubuntu

  • ☁️ capm3 🚀 ck8s 🐧 ubuntu

  • ☁️ capm3 🚀 kadm 🎬 rolling-update-no-wkld 🛠️ ha,misc 🐧 ubuntu

  • ☁️ capm3 🚀 rke2 🎬 wkld-k8s-upgrade 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 ubuntu

  • ☁️ capm3 🚀 rke2 🎬 upgrade-from-prev-release-branch 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 rke2 🛠️ misc,ha 🐧 suse

  • ☁️ capm3 🚀 rke2 🎬 sylva-upgrade 🛠️ ha,misc 🐧 suse

  • ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 ck8s 🎬 rolling-update 🛠️ ha 🐧 ubuntu

  • ☁️ capm3 🚀 rke2|okd 🎬 no-update 🐧 ubuntu|na

  • ☁️ capm3 🚀 rke2 🐧 suse 🎬 upgrade-from-release-1.5

  • ☁️ capm3 🚀 rke2 🐧 suse 🎬 upgrade-to-main

Global config for deployment pipelines

  • autorun pipelines
  • allow failure on pipelines
  • record sylvactl events

Notes:

  • Enabling autorun will make deployment pipelines to be run automatically without human interaction
  • Disabling allow failure will make deployment pipelines mandatory for pipeline success.
  • if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.

Edited by Thomas Morin

Merge request reports

Loading
Loading