GitLab deployed Ingress does not collect any metrics
Summary
Ingress & Prometheus deployed by GitLab into our Kubernetes Cluster does not collect any metrics and does not shows anything in Metrics tab.
Steps to reproduce
- Assign Cluster to Group or Create a new Group Cluster
- Deploy Ingress using the GitLab UI
- Add annotations to Pods
- Add environment to deploy stage in
.gitlab-ci.yml
- Deploy
- Observe non existing metrics
What is the current bug behavior?
Nothing actually happens.
What is the expected correct behavior?
Metrics being visible and collected
Relevant logs and/or screenshots
Ingress is annotated
> kubectl get pod ingress-nginx-ingress-controller-f7bdf6c94-g7c9x -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
Pods are annotated
> kubectl get pod api-prod-679fcfdd9c-mwbws -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
app.gitlab.com/app: infrastructure-api
app.gitlab.com/env: prod
Logs from prometheus-kube-state-metrics
> kubectl -n gitlab-managed-apps logs prometheus-kube-state-metrics-6958b68c64-5n65j
E0921 11:31:11.458754 1 reflector.go:205] k8s.io/kube-state-metrics/collectors/collectors.go:62: Failed to list *v1beta1.Deployment: Get https://10.11.240.1:443/apis/extensions/v1beta1/deployments?limit=500&resourceVersion=0: dial tcp 10.11.240.1:443: i/o timeout
E0921 11:31:11.465480 1 reflector.go:205] k8s.io/kube-state-metrics/collectors/collectors.go:62: Failed to list *v1beta1.ReplicaSet: Get https://10.11.240.1:443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.240.1:443: i/o timeout
E0921 11:31:11.467958 1 reflector.go:205] k8s.io/kube-state-metrics/collectors/collectors.go:62: Failed to list *v1.Job: Get https://10.11.240.1:443/apis/batch/v1/jobs?limit=500&resourceVersion=0: dial tcp 10.11.240.1:443: i/o timeout
E0921 11:31:11.469667 1 reflector.go:205] k8s.io/kube-state-metrics/collectors/collectors.go:62: Failed to list *v1beta1.StatefulSet: Get https://10.11.240.1:443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.240.1:443: i/o timeout
E0921 11:31:11.489574 1 reflector.go:205] k8s.io/kube-state-metrics/collectors/collectors.go:62: Failed to list *v1beta1.DaemonSet: Get https://10.11.240.1:443/apis/extensions/v1beta1/daemonsets?limit=500&resourceVersion=0: dial tcp 10.11.240.1:443: i/o timeout
E0921 11:31:11.516592 1 reflector.go:205] k8s.io/kube-state-metrics/collectors/collectors.go:62: Failed to list *v1.PersistentVolume: Get https://10.11.240.1:443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.240.1:443: i/o timeout
W0929 08:53:38.269956 1 reflector.go:341] k8s.io/kube-state-metrics/collectors/collectors.go:62: watch of *v1.ConfigMap ended with: too old resource version: 18943309 (18943859)
W0929 08:53:42.640570 1 reflector.go:341] k8s.io/kube-state-metrics/collectors/collectors.go:62: watch of *v1.Endpoints ended with: too old resource version: 18943492 (18944543)
W0929 20:37:09.138890 1 reflector.go:341] k8s.io/kube-state-metrics/collectors/collectors.go:62: watch of *v1.ConfigMap ended with: too old resource version: 19127646 (19128728)
W0930 06:00:46.229779 1 reflector.go:341] k8s.io/kube-state-metrics/collectors/collectors.go:62: watch of *v1.ConfigMap ended with: too old resource version: 19274660 (19274970)
Logs from prometheus-server
> kubectl -n gitlab-managed-apps logs prometheus-prometheus-server-774df84bb6-6787w prometheus-server
level=info ts=2019-10-03T09:00:06.83225998Z caller=compact.go:398 component=tsdb msg="write block" mint=1570082400000 maxt=1570089600000 ulid=01DP8DHXQC3QBJ3KEK6RY99KA1
level=info ts=2019-10-03T09:00:07.236637957Z caller=head.go:446 component=tsdb msg="head GC completed" duration=87.81559ms
level=info ts=2019-10-03T09:00:09.535984212Z caller=compact.go:352 component=tsdb msg="compact blocks" count=3 mint=1570060800000 maxt=1570082400000 ulid=01DP8DJ3Z11KBMEKN463TTKMB6 sources="[01DP7RYQZ9RVVFQTGRG6BERNC3 01DP7ZTF7A93ERBG0V428XQ4JG 01DP86P6F96FB6YGAK7XTFJC32]"
level=warn ts=2019-10-03T09:00:11.065350256Z caller=scrape.go:804 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.8.0.24:8080/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
level=warn ts=2019-10-03T09:00:32.135890286Z caller=scrape.go:804 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.8.7.18:8080/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
level=warn ts=2019-10-03T09:00:54.647578111Z caller=scrape.go:804 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.8.0.38:8080/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
Output of checks
This bug happens on GitLab.com
Edited by 🤖 GitLab Bot 🤖