Add Zoekt node, task-state, and stale-storage gauges

Summary

Add three new gauges to lib/gitlab_exporter/database/zoekt.rb to cover Zoekt observability gaps that currently have no equivalent here. These were originally proposed as Rails-side gauges in gitlab-org/gitlab!241045 (merged) but moved here during review because they are pure SQL aggregations and fit the exporter's pattern (DB-derived state metrics).

Background

gitlab-org/gitlab!241045 adds in-process per-request Zoekt metrics to the Rails app (http_zoekt_requests_*, sidekiq_zoekt_requests_*). Those four metrics must live in Rails because they observe data in SafeRequestStore that the exporter cannot see.

The three gauges in this issue are different: they're DB aggregations with no in-process dependency, so they belong here alongside the existing 10 search_zoekt_* metrics in database/zoekt.rb.

Proposed metrics

1. search_zoekt_node_enabled_namespaces

  • Type: gauge

  • Labels: node_id, node_name

  • Description: Number of enabled namespaces per Zoekt node.

  • Query sketch:

    SELECT
  zn.id AS node_id,
  zn.metadata ->> 'name' AS node_name,
  COUNT(DISTINCT zi.zoekt_enabled_namespace_id) AS count
FROM zoekt_nodes zn
LEFT JOIN zoekt_indices zi ON zi.zoekt_node_id = zn.id
GROUP BY zn.id, zn.metadata ->> 'name';

2. search_zoekt_node_tasks

  • Type: gauge

  • Labels: node_id, node_name, state (one of pending, processing, done, skipped, failed, orphaned)

  • Description: Number of Zoekt indexing tasks on a node, broken down by state.

  • Query sketch:

    SELECT
  zn.id AS node_id,
  zn.metadata ->> 'name' AS node_name,
  zt.state,
  COUNT(*) AS count
FROM zoekt_nodes zn
LEFT JOIN zoekt_tasks zt ON zt.zoekt_node_id = zn.id
GROUP BY zn.id, zn.metadata ->> 'name', zt.state;

    The state enum maps (from Search::Zoekt::Taskable):

    Numeric Label
    0 pending
    1 processing
    10 done
    250 skipped
    255 failed
    256 orphaned

    Note that the value of search_zoekt_task_processing_queue_size is approximately sum(search_zoekt_node_tasks{state=~"pending|processing"}) (the existing metric also filters by perform_at <= NOW(), which the new gauge does not). We could either:

    • (a) Add the new gauge alongside the existing one and mark search_zoekt_task_processing_queue_size as deprecated in a follow-up, or
    • (b) Replace the existing one entirely in a single change.

    Option (a) is safer because anyone querying the old metric (alerts, dashboards) keeps working. Recommend going with (a) and removing the old metric in a later release once consumers have migrated.

3. search_zoekt_indices_with_stale_used_storage_bytes

  • Type: gauge

  • Labels: none

  • Description: Number of Zoekt indices whose used_storage_bytes value has not been updated since the last index run.

  • Query sketch:

    SELECT COUNT(*) FROM zoekt_indices
WHERE last_indexed_at >= used_storage_bytes_updated_at;

    This mirrors the Rails Search::Zoekt::Index.with_stale_used_storage_bytes_updated_at scope (defined in ee/app/models/search/zoekt/index.rb).

Implementation notes

Each metric should follow the pattern already used by the other 10 Zoekt metrics in database/zoekt.rb:

  1. SQL constant at the top of ZoektCollector.
  2. Execution in ZoektCollector#run, added to the result hash.
  3. PrometheusMetrics.describe(...) line at the top of ZoektProber.
  4. process_*_results method called from process_results.
  5. add_*_to_metric method that calls @metrics.add(...).
  6. Spec coverage in spec/database/zoekt_spec.rb.

Release / rollout

After this lands, the gem needs to be released and bumped in:

  • gitlab-org/omnibus-gitlab (for Linux package installs)
  • gitlab-org/charts/gitlab (for Helm chart installs)

Until those bumps merge, self-managed installs won't see the new metrics. GitLab.com picks them up on its own cadence.

References