Loki - gateway.basicAuth.username' is required

Job #11395829716 failed for 949a338a:

Installation of Loki Helm chart:

2025/09/18 02:08:13.245499 HelmRelease/loki state changed: InstallFailed - Helm install failed for release loki/loki with chart loki@v3.5.3: execution error at (loki/templates/gateway/secret-gateway.yaml:12:8): 'gateway.basicAuth.username' is required
2025/09/18 02:08:13.319161 HelmRelease/loki state changed: InstallFailed - Helm install failed for release loki/loki with chart loki@v3.5.3: execution error at (loki/templates/gateway/secret-gateway.yaml:12:8): 'gateway.basicAuth.username' is required
Error: Exiting as condition "message" contains ": execution error at " in unit HelmRelease/loki, message: Helm install failed for release loki/loki with chart loki@v3.5.3: execution error at (loki/templates/gateway/secret-gateway.yaml:12:8): 'gateway.basicAuth.username' is required

Loki helm chart values:

  • https://github.com/grafana/loki/blob/v3.5.3/production/helm/loki/values.yaml#L1143

On Sylva, we have a unit that creates initial credential for Loki: https://gitlab.com/sylva-projects/sylva-core/-/blob/main/charts/sylva-units/values.yaml?ref_type=heads#L7947

Then we have a Kyverno ClusterPolicy that we use this secret (+ aggregate with wk-cluster secrets) and patch the Loki helm chart (loki/tenants:) : https://gitlab.com/sylva-projects/sylva-core/-/blob/main/kustomize-units/kyverno-policies/generic/components/loki-aggregated-secrets/loki-aggregated-secrets.yaml?ref_type=heads#L47

There is probably a problem with this KyvernoClusterPolicy, which did not patch/reconcile the helm release at the right time. Investigation is needed

Issue appers randomly in logging nighlty schedule pipeline:

  • https://gitlab.com/sylva-projects/sylva-core/-/jobs/11395829716
  • https://gitlab.com/sylva-projects/sylva-core/-/jobs/11343269070

(prority medium as that break sometimes our CI, but in real usecase/platform the issue should probably resolve itself)

Assignee Loading
Time tracking Loading