[release-1.6] Fix Grafana datasource provisioning during upgrade from release 1.5 to 1.6 (!8722) · Merge requests · Sylva-projects / sylva-core

What does this MR do and why?

This MR fixes an issue encountered while upgrading a management cluster from Sylva release 1.5 to release 1.6. During the upgrade, the Grafana pod enters CrashLoopBackOff with the following datasource provisioning error:

Datasource provisioning error: datasource.yaml config is invalid. Only one datasource per organization can be marked as default

As a result, the monitoring HelmRelease fails and the upgrade remains blocked at the monitoring unit.

Root Cause

The issue is caused by race condition between async event generated by grafana-init unit and monitoring-init unit to reach the grafana deployment deployed in cattle-monitoring-system.

Upgrade from release-1.5 to release-1.6 handing two migrations.

First, Grafana datasource provisioning moves from the Helm-managed datasource configuration provided by the monitoring unit to a Secret-based datasource configuration managed by the new monitoring-init unit.
Second, management of the Grafana Keycloak client moves from keycloak-resources to grafana-init unit. This introduces an asynchronous reconciliation flow that occurs outside the normal sequencing of the Sylva upgrade lifecycle.

Kustomization/keycloak-resources 
└── KeycloakClient/keycloak/grafana-client

Kustomization/grafana-init 
└── Client/keycloak/grafana

The new Crossplane Client/grafana resource creates the new connection Secret keycloak/keycloak-client-secret-grafana-client. External Secrets Operator reads this connection Secret and updates cattle-monitoring-system/grafana-oidc A Kyverno restart policy named restart-deployment-when-oidc-secret-change detects the OIDC Secret update and adds the oidc-secret-update/triggerrestart: xxxx annotation to the Grafana Deployment for triggering a new Grafana deployment rollout.

At the time the new Grafana pod starts, the datasource migration has not yet fully completed. The pod sees both:

the datasource configuration from the previous release, still present in the monitoring helm-managed ConfigMap
the new Secret-based datasource configuration created by monitoring-init.

The grafana-init-sc-datasources init container therefore provisions both datasource definitions into the Grafana datasource directory. Since the resulting configuration contains more than one datasource marked as default, Grafana fails to start with error:

Only one datasource per organization can be marked as default

By the time the monitoring unit is reconciled, the Grafana pod is already in CrashLoopBackOff. This prevents the monitoring HelmRelease from becoming ready and blocks further reconciliation.

Test coverage

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon	Meaning	Available values
☁️	Infra Provider	`capd`, `capo`, `capm3`
🚀	Bootstrap Provider	`kubeadm` (alias `kadm`), `rke2`, `okd`, `ck8s`
🐧	Node OS	`ubuntu`, `suse`, `na`, `leapmicro`
🛠️	Deployment Options	Deployment option list and description
🎬	Pipeline Scenarios	Available scenario list and description
🟢	Enabled units	Any available units name, by default apply to management and workload cluster. Can be prefixed by `mgmt:` or `wkld:` to be applied only to a specific cluster type
🔴	Disabled units	Any available units name, by default apply to management and workload cluster. Can be prefixed by `mgmt:` or `wkld:` to be applied only to a specific cluster type
🏗️	Target platform	Can be used to select specific deployment environment Available platform list and description
⚡	Pipeline control	`autorun`, `manual` or `blocking`. Can be used to override global config and start a deployment pipeline the required way

Global config for deployment pipelines

autorun pipelines
allow failure on pipelines
record sylvactl events

Notes:

Enabling autorun will make deployment pipelines to be run automatically without human interaction
Disabling allow failure will make deployment pipelines mandatory for pipeline success.
if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.

Edited Aug 08, 2026 by Nitin Sharma

[release-1.6] Fix Grafana datasource provisioning during upgrade from release 1.5 to 1.6

What does this MR do and why?

Root Cause

Related reference(s)

Test coverage

CI configuration

Global config for deployment pipelines

Merge request reports