Skip to content

grafana ingress misconfiguration

Summary

The current grafana deployment is not functional on a kubernetes v1.17.3 using the cluster nginx ingress v0.25.1.

It seems that the redirect configuration has changed as of nginx-ingress v0.22. See https://kubernetes.github.io/ingress-nginx/examples/rewrite/#rewrite-target

Steps to reproduce

Install gitlab with grafana enabled. See the configuration yaml below.

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

---
global:
  hosts:
    domain: YOUR.DOMAIN
  smtp:
    enabled: false
  edition: ce
  ingress:
      configureCertmanager: true
      annotations:
          kubernetes.io/ingress.class: nginx
          kubernetes.io/tls-acme: true
      class: nginx
  grafana:
    enabled: true

nginx-ingress:
 enabled: false

Current behavior

The grafana interface fails to load with a redirection loop error. https://gitlab.YOUR.DOMAIN/-/grafana

Also, nothing is configured, no oauth login, no dashboard. I'm not sure it's related.

Expected behavior

Redirection working as expected.

Versions

  • Chart: 3.2.4

  • Platform:

    • Self-hosted: Rancher v2.3.6 on openstack

  • Kubernetes: (kubectl version)

    • Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"archive", BuildDate:"2020-01-25T21:52:51Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
    • Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

  • Helm: (helm version) Not sure, installed through the rancher UI.

Relevant logs

  • grafana pod log entry. See the incorrect path /-/grafana/login seen by the pod. Should be /login 
t=2020-04-21T10:14:11+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/-/grafana/login status=302 remote_addr=127.0.0.1 time_ms=0 size=39 referer=

Fix

This patch should fix this error:

diff --git a/charts/gitlab/charts/gitlab-grafana/templates/ingress.yaml b/charts/gitlab/charts/gitlab-grafana/templates/ingress.yaml
index c27ea2de..63115a85 100644
--- a/charts/gitlab/charts/gitlab-grafana/templates/ingress.yaml
+++ b/charts/gitlab/charts/gitlab-grafana/templates/ingress.yaml
@@ -14,7 +14,7 @@ metadata:
     nginx.ingress.kubernetes.io/proxy-body-size: {{ .Values.ingress.proxyBodySize | quote }}
     nginx.ingress.kubernetes.io/proxy-read-timeout: {{ .Values.ingress.proxyReadTimeout | quote }}
     nginx.ingress.kubernetes.io/proxy-connect-timeout: {{ .Values.ingress.proxyConnectTimeout | quote }}
-    nginx.ingress.kubernetes.io/rewrite-target: /
+    nginx.ingress.kubernetes.io/rewrite-target: /$2
     {{ include "gitlab.certmanager_annotations" . }}
   {{- range $key, $value := merge .Values.ingress.annotations .Values.global.ingress.annotations }}
     {{ $key }}: {{ $value | quote }}
@@ -27,7 +27,7 @@ spec:
           - backend:
               serviceName: {{ .Release.Name }}-grafana
               servicePort: 80
-            path: /-/grafana
+            path: /-/grafana(/|$)(.*)
   {{- if (and $tlsSecret (eq (include "gitlab.ingress.tls.enabled" $) "true" )) }}
   tls:
     - hosts:

However, I'm not sure how older nginx-ingress versions would react to this new rewrite syntax.