### Summary When a relative URL (root prefix) is configured in `gitlab.rb`, the default Prometheus `gitlab-rails` scrape job fails (`up=0`) with a 404 error. This occurs because the `metrics_path` is hardcoded as `/-/metrics` in the `prometheus.yml.erb` Chef template, rather than dynamically respecting the configured relative URL prefix (e.g., `/gitlab/-/metrics`). Because Prometheus connects directly to Puma (port 8080) via Consul service discovery (bypassing NGINX) NGINX rewrite rules cannot mitigate the issue. Furthermore, users cannot override the default `gitlab-rails` job via `prometheus['scrape_configs']` in `gitlab.rb`, as the Chef compiler appends the custom configuration as a duplicate job rather than merging or overwriting the predefined one. ### Steps to reproduce * Configure a relative URL in `/etc/gitlab/gitlab.rb` (e.g., `external_url 'https://example.com/gitlab'`). * Ensure Prometheus monitoring and Consul service discovery are enabled. * Run `sudo gitlab-ctl reconfigure`. * Open the Prometheus UI and navigate to **Status \> Targets**. * Observe the `gitlab-rails` job. ### What is the current _bug_ behavior? The `gitlab-rails` target shows as `DOWN` (`up=0`). Prometheus attempts to scrape `http://<puma-ip>:8080/-/metrics`. Because the Rails application (Puma) is running under the `/gitlab` prefix, it correctly expects `/gitlab/-/metrics` and responds with a `404 Not Found` for the hardcoded, unprefixed path. ### What is the expected _correct_ behavior? The Prometheus Chef template (`/opt/gitlab/embedded/cookbooks/monitoring/templates/default/prometheus.yml.erb`) should dynamically evaluate if a relative URL is configured and prepend it to the `metrics_path` for the `gitlab-rails` job. Alternatively, GitLab Omnibus should allow users to safely disable or merge configurations into the predefined `gitlab-rails` job via the `prometheus['scrape_configs']` array. ### Relevant logs Performing a local curl to the Puma backend port perfectly illustrates the routing mismatch Prometheus is hitting: ```bash # Simulating the exact request the default Prometheus job makes (Fails) root@host:/etc/gitlab# curl http://X.X.X.X:8080/-/metrics Not Found: /-/metrics # Requesting the path with the relative URL prefix (Succeeds) root@host:/etc/gitlab# curl http://X.X.X.X:8080/gitlab/-/metrics # (Returns valid Prometheus metrics) ``` Generated duplicate job collision inside `/var/opt/gitlab/prometheus/prometheus.yml` when attempting to override via `gitlab.rb` : ```yaml # The hardcoded Omnibus job that fails - job_name: gitlab-rails metrics_path: "/-/metrics" consul_sd_configs: - services: - rails # The custom appended job that works, but causes duplicate target issues - job_name: gitlab-rails metrics_path: "/gitlab/-/metrics" consul_sd_configs: - services: - rails ``` <details> <summary> <pre> </pre> </summary></details> ### Details of package version <details> <summary>Provide the package version installation details</summary> <pre> <!--For Debian, Ubuntu based systems, paste the output of: `dpkg-query -l "gitlab-*"` <p data-sourcepos="51:1-52:25">For RHEL based systems (CentOS, RHEL, OL, Scientific, OpenSUSE, SLES), paste the output of: <code data-sourcepos="52:1-52:25">rpm -qa | grep 'gitlab'</code></p> <p data-sourcepos="54:1-55:3">If you can't provide these details, see <a data-sourcepos="54:41-54:78" href="https://about.gitlab.com/getting-help/">https://about.gitlab.com/getting-help/</a> page on where to ask your question. --&gt;</p> </pre> </details> <h3 id="user-content-environment-details" data-sourcepos="60:1-60:23">Environment details<a href="#environment-details" aria-label="Link to heading 'Environment details'" data-heading-content="Environment details" class="anchor"></a></h3> <ul data-sourcepos="62:1-75:40"> <li data-sourcepos="62:1-62:42">Operating System: <code data-sourcepos="62:21-62:42">REPLACE-WITH-DETAILS</code></li> <li data-sourcepos="63:1-66:33">Installation Target, remove incorrect values: <ul data-sourcepos="64:3-66:33"> <li data-sourcepos="64:3-64:22">Bare Metal Machine</li> <li data-sourcepos="65:3-65:68">VM: Digital Ocean, AWS, GCP, Azure, Other <code data-sourcepos="65:47-65:68">REPLACE-WITH-DETAILS</code></li> <li data-sourcepos="66:3-66:33">Other: <code data-sourcepos="66:12-66:33">REPLACE-WITH-DETAILS</code></li> </ul> </li> <li data-sourcepos="67:1-70:33">Installation Type, remove incorrect values: <ul data-sourcepos="68:3-70:33"> <li data-sourcepos="68:3-68:20">New Installation</li> <li data-sourcepos="69:3-69:47">Upgrade from version <code data-sourcepos="69:26-69:47">REPLACE-WITH-DETAILS</code></li> <li data-sourcepos="70:3-70:33">Other: <code data-sourcepos="70:12-70:33">REPLACE-WITH-DETAILS</code></li> </ul> </li> <li data-sourcepos="71:1-71:76">Is there any other software running on the machine: <code data-sourcepos="71:55-71:76">REPLACE-WITH-DETAILS</code></li> <li data-sourcepos="72:1-72:49">Is this a single or multiple node installation?</li> <li data-sourcepos="73:1-76:0">Resources <ul data-sourcepos="74:3-75:40"> <li data-sourcepos="74:3-74:31">CPU: <code data-sourcepos="74:10-74:31">REPLACE-WITH-DETAILS</code></li> <li data-sourcepos="75:3-75:40">Memory total: <code data-sourcepos="75:19-75:40">REPLACE-WITH-DETAILS</code></li> </ul> </li> </ul> <h3 id="user-content-configuration-details" data-sourcepos="77:1-77:25">Configuration details<a href="#configuration-details" aria-label="Link to heading 'Configuration details'" data-heading-content="Configuration details" class="anchor"></a></h3> <details> <summary> Provide the relevant sections of `/etc/gitlab/gitlab.rb` </summary> <pre> <!-- Paste the details but omit lines starting with `#`. Relevant lines can be retrieved by running `grep -v -e '^#' -e '^$' /etc/gitlab/gitlab.rb`. Be certain to sanitize sensitive configuration such as passwords and secrets.--> </pre> </details> ``` hi gitlab-ee 18.11.4-ee.0 amd64 GitLab Enterprise Edition (including NGINX, Postgres, Redis) ``` ### Environment details * Operating System: Ubuntu 22.04 * Installation Target, remove incorrect values: * VM: on-prem * Installation Type, remove incorrect values: * Cluster deployed with GET * Is there any other software running on the machine: no * Is this a single or multiple node installation? 3k reference architecture cluster