Scrape gitlab-rails metrics under the relative URL root

What does this MR do?

When external_url includes a path, for example https://gitlab.example.com/gitlab, Puma serves the Rails application under that path, so the Rails metrics endpoint is /gitlab/-/metrics. The gitlab-rails job that rails_scrape_configs writes into prometheus.yml still asks for /-/metrics, Puma answers 404, and the target stays down, as the issue reports.

The job takes two shapes, and the path comes from a different node in each:

• The static_configs job scrapes the Puma on the same node, so its metrics_path is built from that node's gitlab_rails['gitlab_relative_url'].
• The consul_sd_configs job, used with consul['monitoring_service_discovery'] = true, is written on the node that runs Prometheus. That node may run nothing but the monitoring role, and its own settings say nothing about the Rails nodes it discovers, which need not even share one prefix. So each Rails node publishes its own metrics path in the metadata of its rails Consul service, and the job copies __meta_consul_service_metadata_metrics_path into __metrics_path__ with a relabel rule. The job keeps metrics_path: "/-/metrics", so a Rails node that publishes no path, for example one still on an older package, is scraped as before.

Without a relative URL nothing changes: the path stays /-/metrics.

Both shapes build the path in one helper, Prometheus.rails_metrics_path, which the Puma recipe calls for the Consul metadata too.

Why gitlab_rails['gitlab_relative_url']:

• GitlabRails.parse_external_url sets it from the path of external_url, with a trailing slash removed, and only when the path is neither empty nor / (gitlab_rails.rb lines 187-189). Without a relative URL it stays nil, and the path is /-/metrics.
• It is the path Puma serves Rails under: the gitlab-rails recipe exports it as RAILS_RELATIVE_URL_ROOT (gitlab-rails.rb lines 612-613), and GitLab's config.ru mounts the application there (map ENV['RAILS_RELATIVE_URL_ROOT'].presence || "/").
• gitlab_rails is parsed before prometheus: generate_config calls each block's parse_variables in priority order (settings_dsl.rb lines 206-209), and package/libraries/config/gitlab.rb gives gitlab_rails priority 15 (line 104) and prometheus priority 21 (line 83). So the value is set when the scrape configs are built. GitlabHealthcheckHelper reads the same attribute the same way to build the healthcheck URL.

For the Consul job, the shape follows what is already in the cookbooks: the kubernetes-pods job in the same library copies a pod annotation into __metrics_path__ with the same relabel rule (prometheus.rb lines 672-675), and Gitaly and Praefect already pass meta to consul_service (gitaly/recipes/enable.rb line 201, praefect/recipes/enable.rb line 101). The Puma one passed none, so puma['consul_service_meta'], documented in gitlab.rb.template, never reached Consul. The rails service now carries it, merged after metrics_path, and the template says so. A monitoring node set up as in Standalone Prometheus using the Linux package needs no new setting.

Two details of the path, both in the helper:

• A gitlab_rails['gitlab_relative_url'] set directly in gitlab.rb skips the chomp("/") in parse_external_url, so slashes at either end are trimmed before /-/metrics is appended.
• The prefix is decoded. Prometheus puts __metrics_path__ into the path of the scrape URL and escapes it again (scrape/target.go lines 248-252 of 3.13.2), so with external_url 'https://gitlab.example.com/git%20lab' the job asked for /git%2520lab/-/metrics, which Puma answers with a 404. A prefix that is not a valid percent-encoding, such as /100% or /%ff, is kept as written, because decoding it would raise or produce bytes that are not UTF-8 and stop reconfigure.

The issue also reports that a prometheus['scrape_configs'] entry named gitlab-rails is added next to the default job instead of replacing it. This MR does not change that: it is a separate request, tracked in #8709.

How to validate locally

1. In /etc/gitlab/gitlab.rb, set a relative URL:

   external_url 'http://gitlab.example.com/gitlab'

2. Run sudo gitlab-ctl reconfigure.

3. Run sudo grep -A1 'job_name: gitlab-rails' /var/opt/gitlab/prometheus/prometheus.yml. Before this change it shows metrics_path: "/-/metrics"; with it, metrics_path: "/gitlab/-/metrics".

4. Run curl -s http://localhost:9090/api/v1/query --data-urlencode 'query=up{job="gitlab-rails"}'. Before this change the value is 0; with it, 1 once Prometheus has scraped the target.

With Consul service discovery (consul['enable'] = true and consul['monitoring_service_discovery'] = true), on a separate monitoring node and Rails nodes under a relative URL:

1. On a Rails node with external_url 'http://gitlab.example.com/gitlab', run sudo cat /var/opt/gitlab/consul/config.d/rails-service.json. With this change it carries "meta":{"metrics_path":"/gitlab/-/metrics"}.
2. On the monitoring node, run curl -s 'http://localhost:9090/api/v1/targets?scrapePool=gitlab-rails'. Each Rails target's scrapeUrl ends in the path its node published, and its health is up.

Specs before and after

In ruby:3.3.12 (.ruby-version) with the project's bundle, on trees built with git archive from the commits. The branch's three spec files on 077fb6b3f173, before this MR:

$ bundle exec rspec spec/chef/cookbooks/monitoring/libraries/prometheus_spec.rb spec/chef/cookbooks/monitoring/recipes/prometheus_spec.rb spec/chef/cookbooks/gitlab/recipes/puma_spec.rb
  1) Prometheus.rails_metrics_path relative_url: nil, metrics_path: "/-/metrics" returns the path Puma serves the Rails metrics at
     NoMethodError:
       undefined method `rails_metrics_path' for module Prometheus
 12) monitoring::prometheus when GitLab is served under a relative URL scrapes the Rails metrics under the relative URL
       -:metrics_path => "/gitlab/-/metrics",
       +:metrics_path => "/-/metrics",
 15) monitoring::prometheus with Consul service discovery scrapes each Rails node at the metrics path it publishes in Consul
       -:relabel_configs => [{:action=>"replace", :regex=>"(.+)", :source_labels=>["__meta_consul_service_metadata_metrics_path"], :target_label=>"__metrics_path__"}],
 18) gitlab::puma with Ubuntu 16.04 with Consul service discovery publishes the Rails metrics path in the service metadata
       meta {"metrics_path"=>"/-/metrics"}, was nil
96 examples, 21 failures

The 21 failures are every new example: the helper table, the three static examples, the three Consul examples and the four Puma examples. On the first commit alone the same files give 96 examples, 19 failures: the static examples under /gitlab and /a/b pass there, and the rest fail as above. The static example under /gitlab uses external_url 'https://gitlab.example.com/gitlab/', with a trailing slash, so it also holds that no // ends up in the path.

With both commits, the three spec files, every spec under the monitoring and Consul cookbooks with the Puma recipe's, the gitlab-rails recipe's spec, and RuboCop on the five Ruby files:

$ bundle exec rspec spec/chef/cookbooks/monitoring/libraries/prometheus_spec.rb spec/chef/cookbooks/monitoring/recipes/prometheus_spec.rb spec/chef/cookbooks/gitlab/recipes/puma_spec.rb
96 examples, 0 failures
$ bundle exec rspec spec/chef/cookbooks/monitoring spec/chef/cookbooks/consul spec/chef/cookbooks/gitlab/recipes/puma_spec.rb
344 examples, 0 failures
$ bundle exec rspec spec/chef/cookbooks/gitlab/recipes/gitlab-rails_spec.rb
202 examples, 0 failures
$ bundle exec rubocop files/gitlab-cookbooks/monitoring/libraries/prometheus.rb files/gitlab-cookbooks/gitlab/recipes/puma.rb spec/chef/cookbooks/monitoring/libraries/prometheus_spec.rb spec/chef/cookbooks/monitoring/recipes/prometheus_spec.rb spec/chef/cookbooks/gitlab/recipes/puma_spec.rb
5 files inspected, no offenses detected

The steps above on a 19.4.2 package

I ran the validation steps in a gitlab/gitlab-ce:latest container (19.4.2), with external_url 'http://gitlab.example.com/gitlab'. The image's monitoring/libraries/prometheus.rb is byte-identical to the one on 077fb6b3f173, so replacing it with the first commit's file and reconfiguring applies exactly that commit's change.

With the packaged file, Puma answers the path the job asks for with a 404 and the prefixed path with a 200, and the target is down:

- job_name: gitlab-rails
  metrics_path: "/-/metrics"
/-/metrics -> 404
/gitlab/-/metrics -> 200
{"health":"down","scrapeUrl":"http://127.0.0.1:8080/-/metrics","lastError":"server returned HTTP status 404 Not Found"}
up{job="gitlab-rails"} 0

With the first commit's file, gitlab-ctl reconfigure updated 2 of 824 resources, prometheus.yml and the Prometheus reload, and the target came up:

- job_name: gitlab-rails
  metrics_path: "/gitlab/-/metrics"
{"health":"up","scrapeUrl":"http://127.0.0.1:8080/gitlab/-/metrics","lastError":""}
up{job="gitlab-rails"} 1

This ran on the first commit. For this external_url the second commit renders the same static job, which the specs hold.

Consul service discovery with Consul 1.22.7 and Prometheus 3.13.2

The versions this repository ships, in containers, with the files the cookbooks render (ChefSpec on this branch, with step_into for consul_service):

• monitor: a Consul server, with Prometheus in its network namespace so consul_sd_configs reaches the local agent as on a monitoring node. Its prometheus.yml is the one rendered for the Standalone Prometheus node of the docs: roles ['monitoring_role'], external_url 'http://gitlab.example.com', prometheus['monitor_kubernetes'] = false, Consul service discovery on.
• rails-a, rails-c: a Consul client agent with the rails-service.json rendered for external_url 'http://gitlab.example.com/gitlab' and 'http://gitlab.example.com/git%20lab', and Puma 8.0.2 with Rack 2.2.23 (the versions in GitLab's Gemfile.lock) serving a stand-in app mounted the way GitLab's config.ru mounts Rails, under RAILS_RELATIVE_URL_ROOT.
• rails-b: the same at the root, with the rails-service.json the previous commit renders, which has no metadata.

The one edit to a rendered service definition is dropping address, as consul_service does when advertise_addr is set, so each service takes its agent's address instead of 127.0.0.1.

With the prometheus.yml the previous commit renders:

{"node":"rails-a","scrapeUrl":"http://172.24.0.3:8080/-/metrics","health":"down","lastError":"server returned HTTP status 404 Not Found"}
{"node":"rails-b","scrapeUrl":"http://172.24.0.4:8080/-/metrics","health":"up","lastError":""}
{"node":"rails-c","scrapeUrl":"http://172.24.0.5:8080/-/metrics","health":"down","lastError":"server returned HTTP status 404 Not Found"}

With the one this branch renders, which only adds the relabel rule:

{"node":"rails-a","scrapeUrl":"http://172.24.0.3:8080/gitlab/-/metrics","health":"up","lastError":""}
{"node":"rails-b","scrapeUrl":"http://172.24.0.4:8080/-/metrics","health":"up","lastError":""}
{"node":"rails-c","scrapeUrl":"http://172.24.0.5:8080/git%20lab/-/metrics","health":"up","lastError":""}

The Consul catalog for the rails service, the same in both runs:

{"Node":"rails-a","ServiceAddress":"","ServicePort":8080,"ServiceMeta":{"metrics_path":"/gitlab/-/metrics"}}
{"Node":"rails-b","ServiceAddress":"","ServicePort":8080,"ServiceMeta":{}}
{"Node":"rails-c","ServiceAddress":"","ServicePort":8080,"ServiceMeta":{"metrics_path":"/git lab/-/metrics"}}

Related issues

Closes #9933

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• MR title and description are up to date, accurate, and descriptive.
• MR targeting the appropriate branch.
• Latest Merge Result pipeline is green.
• When ready for review, MR is labeled workflowready for review per the Distribution MR workflow.
• The UBT version and corresponding checksum hash have been updated and referenced in the merge request if applicable. Not applicable: no software definition changes.
  • UBT EE pipeline (Trigger:ee-package-ubt) is green

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

• The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
  • To debug QA failures, refer to the Investigate QA failures section.
• If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
• If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
• If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes. The steps are under How to validate locally. The single-node ones pass on a 19.4.2 package with the first commit, and the Consul ones were run with Consul 1.22.7 and Prometheus 3.13.2 on the files the cookbooks render (details above).
• Documentation created/updated. gitlab.rb.template says what the rails Consul service now carries next to puma['consul_service_meta']. No page needs a change: a monitoring node needs no new setting.
• Tests added.
• Integration tests added to GitLab QA. Not added: the spec covers the rendered scrape configuration.
• Equivalent MR/issue for the GitLab Chart opened. Not applicable: this is the Linux package's bundled Prometheus.
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file. Not applicable: no new setting.