Skip to content

Installing prometheus in newly created kubernetes cluster throws ConfigMap error

Summary

Running Gitlab EE 11.0.2.

After having created a new Kubernetes cluster and installing Tiller, when trying to install Prometheus the following error is displayed:

Error: release prometheus failed: ConfigMap in version "v1" cannot be handled as a ConfigMap: v1.ConfigMap: Data: ReadString: expects " or n, parsing 37 ..."alerts":{... at {"apiVersion":"v1","data":{"alerts":{},"prometheus.yml":{"rule_files":["/etc/config/rules","/etc/config/alerts"],"scrape_configs":[{"job_name":"prometheus","static_configs":[{"targets":["localhost:9090"]}]},{"bearer_token_file":"/var/run/secrets/kubernetes.io/serviceaccount/token","job_name":"kubernetes-cadvisor","kubernetes_sd_configs":[{"role":"node"}],"metric_relabel_configs":[{"regex":"(.+)-.+-.+","source_labels":["pod_name"],"target_label":"environment"}],"relabel_configs":[{"action":"labelmap","regex":"__meta_kubernetes_node_label_(.+)"},{"replacement":"kubernetes.default.svc:443","target_label":"__address__"},{"regex":"(.+)","replacement":"/api/v1/nodes/${1}/proxy/metrics/cadvisor","source_labels":["__meta_kubernetes_node_name"],"target_label":"__metrics_path__"}],"scheme":"https","tls_config":{"ca_file":"/var/run/secrets/kubernetes.io/serviceaccount/ca.crt","insecure_skip_verify":true}},{"job_name":"kubernetes-service-endpoints","kubernetes_sd_configs":[{"role":"endpoints"}],"relabel_configs":[{"action":"keep","regex":true,"source_labels":["__meta_kubernetes_service_annotation_prometheus_io_scrape"]},{"action":"replace","regex":"(https?)","source_labels":["__meta_kubernetes_service_annotation_prometheus_io_scheme"],"target_label":"__scheme__"},{"action":"replace","regex":"(.+)","source_labels":["__meta_kubernetes_service_annotation_prometheus_io_path"],"target_label":"__metrics_path__"},{"action":"replace","regex":"(.+)(?::\\d+);(\\d+)","replacement":"$1:$2","source_labels":["__address__","__meta_kubernetes_service_annotation_prometheus_io_port"],"target_label":"__address__"},{"action":"labelmap","regex":"__meta_kubernetes_service_label_(.+)"},{"action":"replace","source_labels":["__meta_kubernetes_namespace"],"target_label":"kubernetes_namespace"},{"action":"replace","source_labels":["__meta_kubernetes_service_name"],"target_label":"kubernetes_name"}]},{"bearer_token_file":"/var/run/secrets/kubernetes.io/serviceaccount/token","job_name":"kubernetes-nodes","kubernetes_sd_configs":[{"role":"node"}],"metric_relabel_configs":[{"regex":"(.+)-.+-.+","source_labels":["pod_name"],"target_label":"environment"}],"relabel_configs":[{"action":"labelmap","regex":"__meta_kubernetes_node_label_(.+)"},{"replacement":"kubernetes.default.svc:443","target_label":"__address__"},{"regex":"(.+)","replacement":"/api/v1/nodes/${1}/proxy/metrics","source_labels":["__meta_kubernetes_node_name"],"target_label":"__metrics_path__"}],"scheme":"https","tls_config":{"ca_file":"/var/run/secrets/kubernetes.io/serviceaccount/ca.crt","insecure_skip_verify":true}},{"bearer_token_file":"/var/run/secrets/kubernetes.io/serviceaccount/token","job_name":"kubernetes-pods","kubernetes_sd_configs":[{"role":"pod"}],"relabel_configs":[{"action":"keep","regex":"true","source_labels":["__meta_kubernetes_pod_annotation_prometheus_io_scrape"]},{"action":"replace","regex":"(.+)","source_labels":["__meta_kubernetes_pod_annotation_prometheus_io_path"],"target_label":"__metrics_path__"},{"action":"replace","regex":"([^:]+)(?::[0-9]+)?;([0-9]+)","replacement":"$1:$2","source_labels":["__address__","__meta_kubernetes_pod_annotation_prometheus_io_port"],"target_label":"__address__"},{"action":"labelmap","regex":"__meta_kubernetes_pod_label_(.+)"},{"action":"replace","source_labels":["__meta_kubernetes_namespace"],"target_label":"kubernetes_namespace"},{"action":"replace","source_labels":["__meta_kubernetes_pod_name"],"target_label":"kubernetes_pod_name"}],"tls_config":{"ca_file":"/var/run/secrets/kubernetes.io/serviceaccount/ca.crt","insecure_skip_verify":true}}]},"rules":{}},"kind":"ConfigMap","metadata":{"labels":{"app":"prometheus-prometheus","chart":"prometheus-2.0.0","component":"server","heritage":"Tiller","release":"prometheus"},"name":"prometheus-server","namespace":"gitlab-managed-apps"}}

Steps to reproduce

  1. Create new cluster through Gitlab.
  2. Install Tiller
  3. Install Prometheus

What is the current bug behavior?

Prometheus won't install and throws the error above.

What is the expected correct behavior?

Prometheus installs correctly.

Relevant logs and/or screenshots

See above.