Scrape Praefect metrics from the Pods, not the Service

José M. Requena Plensrequested to merge
gitlab-community/gitlab-org/charts/gitlab:jmrp-charts-praefect-metrics-scrape into master

What does this MR do?

The Praefect Service carries Prometheus scrape annotations whose port comes from .Values.metrics.metricsPort, a key the Praefect chart does not define, so with the default values the Service renders gitlab.com/prometheus_port: "" and prometheus.io/port: "". The Praefect Pods carry the same annotations with metrics.port (9236) and metrics.path, which are right. Everything else in the Praefect chart that names the metrics port reads metrics.port as well: the listen address in the ConfigMap and in PRAEFECT_PROMETHEUS_LISTEN_ADDR, the container port and the http-metrics port of the Service. metricsPort is a key of the Gitaly chart only, kept there as a deprecated alias of Gitaly's own metrics.port.

I installed the chart in a kind cluster with Praefect and the bundled Prometheus to see what that does. The kubernetes-service-endpoints job makes a target of every port of an annotated Service, and with an empty port annotation nothing rewrites the address, so it scraped port 8075 of each Praefect Pod as well as 9236. Port 8075 is the gRPC listener, so those targets were down, and the metrics of each Pod were collected twice: once by the kubernetes-pods job through the Pod annotations and once by the kubernetes-service-endpoints job through the Service. The Service annotations are also rendered with metrics.serviceMonitor.enabled: true, while the parameter table says enabling the ServiceMonitor removes the prometheus.io scrape annotations.

This MR removes the scrape annotations from the Praefect Service. The Pods keep theirs, and the ServiceMonitor still points at the http-metrics port of the Service, so each Praefect is scraped once, on its metrics port. !1947 (merged) made the same move for Gitaly, from its Service to its Pods, and every other GitLab component of the chart that carries scrape annotations (Gitaly, Webservice, Sidekiq, GitLab Shell, GitLab Pages, KAS, GitLab Exporter, GitLab Runner and the registry) carries them on its Pods only. The Praefect Pod annotations came with the ServiceMonitor support in d5618996, which left the Service ones in place. The MR also adds an upgrade note for the change in behavior below to doc/releases/10_0.md, under 10.6.0.

I also tried correcting the key to .Values.metrics.port instead. All targets were up then, but each Praefect was still scraped by both jobs, with every series collected twice, so I went with removing the annotations.

One change in behavior: a Prometheus that discovers targets only from Service annotations, and not from Pod annotations, no longer scrapes Praefect. It scraped it before through the Service's http-metrics port, alongside the failing gRPC targets, and it does not scrape Gitaly or the other components listed above either. With the bundled Prometheus, the kubernetes-pods job still scrapes Praefect and the kubernetes-service-endpoints job no longer does, so a query that selects Praefect series by that job stops matching. The upgrade note says this, and what to configure instead: Pod discovery, or the Praefect ServiceMonitor with the Prometheus Operator.

Rendered Service annotations

helm template with Helm 4.3.0, the default values from spec/helm_template_helper.rb and global.praefect.enabled: true.

On master (c27d29254), with the default values and with gitlab.praefect.metrics.serviceMonitor.enabled=true:

metadata:
  name: test-praefect
  annotations:
    gitlab.com/prometheus_scrape: "true"
    gitlab.com/prometheus_port: ""
    prometheus.io/scrape: "true"
    prometheus.io/port: ""

The port the Service annotations name on master, against the port Praefect listens on (prometheus_listen_addr, PRAEFECT_PROMETHEUS_LISTEN_ADDR, the container and Service http-metrics ports and, without a ServiceMonitor, the Pod annotations, which agree in every case):

Values Praefect listens on Service port annotations on master
defaults 9236 ""
gitlab.praefect.metrics.port=9999 9999 ""
gitlab.praefect.metrics.metricsPort=9999 9236 "9999"
gitlab.praefect.metrics.serviceMonitor.enabled=true 9236 ""

With this MR the Service has no scrape annotations in any of these cases. Across the whole render, the four annotation lines of the Praefect Service are the only difference from master, apart from the random suffix of the Webservice test runner Pod name.

Cluster test with the bundled Prometheus

kind v0.33.0 with Kubernetes v1.37.0, the chart at fdcf0c3e9 with global.gitlabVersion: 19.4.1, global.praefect.enabled: true (two Praefect replicas, three Gitaly replicas) and prometheus.install: true. Webservice, Sidekiq, GitLab Shell, the toolbox, the migrations, GitLab Exporter, Mailroom, the registry, KAS, MinIO, NGINX Ingress and GitLab Runner were disabled, Praefect used a PostgreSQL 16 Deployment in the same namespace, and I created the wildcard CA Secret the Pods mount by hand, since this trimmed install did not create one. I installed master, upgraded to master with only the key corrected, then upgraded to this branch, and after each step read the Prometheus targets through the API server's service proxy once two scrape intervals had passed. I did not run the Prometheus Operator; the ServiceMonitor renders the same as on master.

master:

Job Scrape URL Health
kubernetes-pods http://10.244.0.12:9236/metrics up
kubernetes-pods http://10.244.0.13:9236/metrics up
kubernetes-service-endpoints http://10.244.0.12:8075/metrics down
kubernetes-service-endpoints http://10.244.0.12:9236/metrics up
kubernetes-service-endpoints http://10.244.0.13:8075/metrics down
kubernetes-service-endpoints http://10.244.0.13:9236/metrics up

The two down targets reported net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x06\x04...", an HTTP/2 frame from the gRPC listener. Both jobs scraped the same two 9236 endpoints, so every Praefect series was stored once under each job.

master with metricsPort replaced by port in the Service annotations: four targets, all up, two per job, both jobs on the same two 9236 endpoints.

This branch:

Job Scrape URL Health
kubernetes-pods http://10.244.0.12:9236/metrics up
kubernetes-pods http://10.244.0.13:9236/metrics up

Only kubernetes-pods scrapes Praefect. The branch now sits on c27d29254 with the same template change.

Specs and checks

The MR adds no specs: the change removes template lines and adds no logic. The existing specs, on this branch, in Ruby 3.3.12 with Helm 4.3.0 and gomplate v5.0.0 (the versions in .gitlab-ci.yml and scripts/ci/install_spec_dependencies.sh):

  • praefect_spec.rb, monitoring_spec.rb, annotations_spec.rb, traffic_distribution_spec.rb, gitaly_spec.rb and labels_spec.rb: 144 examples, 0 failures.
  • The other specs that render Praefect (cluster_domain, dns_config, global, image_tag, node_selector, priorityclass, securitycontext, shared_secrets, strategy, tolerations, topology_spread_constraints and spec/integration/check_config/gitaly_spec.rb): 193 examples, 0 failures.

The upgrade note, with the commands of the check_docs_content, check_docs_markdown and check_docs_links jobs in their image: Vale at error level, 0 alerts in 127 files; markdownlint-cli2, 0 issues in 127 files; lychee, 0 errors. Vale at every level reports nothing on the new section.

Related issues

Closes #6454 (closed)

Author checklist

For general guidance, please follow our Contributing guide.

Required

For anything in this list which will not be completed, please provide a reason in the MR discussion.

  • Merge Request Title and Description are up to date, accurate, and descriptive.
  • MR targeting the appropriate branch.
  • MR has a green pipeline.
  • Documentation created/updated. An upgrade note in doc/releases/10_0.md. The metrics.serviceMonitor.enabled row already says that enabling it removes the scrape annotations, which is now true of the Service too, and the Prometheus section of doc/installation/tools.md lists Praefect on 9236, the port its Pods are annotated with.
  • Tests added/updated. None added, as agreed in review: the change removes template lines only, and the existing specs that render Praefect pass.
  • Have you validated that your change works end-to-end on a real cluster deployment?
    • In a kind cluster, described under "Cluster test with the bundled Prometheus" above.
  • Equivalent MR/issue for omnibus-gitlab opened. Not applicable: Omnibus does not render Kubernetes Services.
  • When ready for review, MR is marked as ready (not a draft) and assigned to a reviewer.

Reviewers checklist

Edited by Clemens Beck

Merge request reports