Expose the GitLab Exporter Zoekt probe so search_zoekt_* metrics are available on GitLab Self-Managed

Problem to solve

The 13 database-derived search_zoekt_* gauges are documented for GitLab Self-Managed in the Prometheus metrics reference, but no GitLab Self-Managed installation can emit them, because the GitLab Exporter Zoekt probe is never referenced in the generated exporter configuration.

Administrators setting up exact code search have no Prometheus signal for Zoekt node health, index state, or indexing task backlog.

Reported by a customer running GitLab 19.4.3 on the Helm chart (internal ticket, Ultimate).

Verified findings

The probe code ships in the image. Database::ZoektProber and ZoektCollector are defined in lib/gitlab_exporter/database/zoekt.rb and are present at gem tag 17.0.2, which is the appVersion pinned by charts/gitlab/charts/gitlab-exporter/Chart.yaml at chart v10.4.3 (GitLab 19.4.3).

The chart configuration never references it. charts/gitlab/charts/gitlab-exporter/templates/configmap.yaml renders the database, database_bloat, sidekiq, ruby and metrics probe groups, covering ci_builds, tuple_stats, rows_count, pg_sequences and optionally geo_node_status. There is no zoekt entry in either the database or the metrics group, and values.yaml exposes no key to add one. Because the config is mounted from the ConfigMap with subPath, extraVolumeMounts cannot substitute a replacement file.

The Linux package has the same gap. files/gitlab-cookbooks/monitoring/templates/gitlab-exporter.yml.erb omits the probe at tag 19.4.3+ee.0, so this is not specific to the chart.

Probe group names become URL paths. WebExporter.mount_probes calls @server.mount_proc("/#{probe_name}"), so a probe placed in the database group is served on /database, while the chart's ServiceMonitor scrapes metrics.path: /metrics. Placement therefore determines whether a default scrape picks the metrics up.

Proposal

  1. Add the Zoekt probe to the rendered exporter configuration:

    zoekt:
  class_name: Database::ZoektProber
  <<: *db_common

  2. Gate it behind a values key, mirroring the existing probeGeo pattern, so installations without Zoekt do not pay for the queries. For example probeZoekt, default false.

  3. Place it in the metrics probe group, or document the extra /database scrape target, so that a default Prometheus configuration collects the metrics.

  4. Mirror the change in gitlab-org/omnibus-gitlab so both install methods behave the same.

  5. Update the documentation note added by gitlab-org/gitlab#603839 (closed) once the probe is available.

Notes

The probe queries zoekt_nodes, zoekt_indices, zoekt_repositories and zoekt_tasks. Worth checking the cost of ZOEKT_INDICES_STORAGE_QUERY and ZOEKT_NODE_TASKS_QUERY on large instances before enabling by default, which is a further argument for an opt-in value.

GitLab.com appears to configure this probe through its own infrastructure configuration rather than through these templates, which would explain why the gap went unnoticed. See the rollout checklist in gitlab-org/gitlab#507875 (closed).

References