[release-1.6] Upgrade from release 1.5 to 1.6 gets stuck at the monitoring unit - CAPO Deployment

Summary

During an upgrade from Sylva release 1.5 to release 1.6 in CAPO pipelines, the deployment becomes stuck at the monitoring unit.

The monitoring Kustomization reports a failed HelmRelease because the Grafana pod does not become ready:

monitoring False health check failed: HelmRelease/sylva-system/monitoring status: 'Failed'

The Grafana pod enters CrashLoopBackOff with the following datasource provisioning error:

Datasource provisioning error: datasource.yaml config is invalid. Only one datasource per organization can be marked as default

Logs from CI:

2026/07/29 06:31:48.446665 Unit timeout exceeded: unit Kustomization/monitoring did not became ready after 15m0s
Details on Kustomization/monitoring and related resources:
IDENTIFIER                                     STATUS     REASON              MESSAGE
Kustomization/sylva-system/monitoring          InProgress                     Kustomization generation is 3, but latest observed generation is 2
╰┄╴HelmRelease/sylva-system/monitoring         InProgress                     Running 'upgrade' action with timeout of 5m0s
   ├┄╴┬┄┄[Conditions]
   ┆  ├┄╴Reconciling                           True       Progressing         Running 'upgrade' action with timeout of 5m0s
   ┆  ├┄╴Ready                                 Unknown    Progressing         Running 'upgrade' action with timeout of 5m0s
   ┆  ╰┄╴Released                              False      UpgradeFailed       Helm upgrade failed for release cattle-monitoring-system/rancher-monitoring with chart rancher-monitoring@107.2.2+up69.8.2-rancher.26: context deadline exceeded
   ╰┄╴┬┄┄[Events]
      ├┄╴2026-07-29 05:08:03                   Normal     HelmChartCreated    Created HelmChart/sylva-system/sylva-system-monitoring with SourceRef 'HelmRepository/sylva-system/unit-monitoring'
      ├┄╴2026-07-29 05:08:04                   Normal     HelmChartConfigured Configured HelmChart/sylva-system/sylva-system-monitoring with SourceRef 'HelmRepository/sylva-system/unit-monitoring'

Job:

related references

Details

Edited by Nitin Sharma