Fix missing cadvisor metrics
It looks like in k8s 1.7, metrics from cadvisor
were moved into their own scrape endpoint. This explains why when running on cluster 1.7 or higher, no CPU or Memory metrics are found.
We need to add an additional scrape job, the one below works for me:
- job_name: kubernetes-cadvisor
scrape_interval: 15s
scrape_timeout: 15s
metrics_path: /metrics
scheme: https
kubernetes_sd_configs:
- api_server: https://kubernetes.default.svc:443
role: node
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
namespaces:
names: []
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
relabel_configs:
- source_labels: []
separator: ;
regex: __meta_kubernetes_node_label_(.+)
replacement: $1
action: labelmap
- source_labels: []
separator: ;
regex: (.*)
target_label: __address__
replacement: kubernetes.default.svc:443
action: replace
- source_labels: [__meta_kubernetes_node_name]
separator: ;
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
action: replace
metric_relabel_configs:
- source_labels: [pod_name]
separator: ;
regex: (.+)-.+-.+
target_label: environment
replacement: $1
action: replace
However there is a note that add this blindly to pre-1.7 k8s instances will cause duplication of metrics. We need to think of a solution for that, somehow.