Log Zoekt enabled namespaces without replicas

Ravi Kumarrequested to merge
zoekt-missing-indices-metric into master

What does this MR do and why?

Logs the number of Zoekt-enabled namespaces that have no replica, once per minute, so the rollout backlog can be charted over time in Kibana.

Today the backlog is only visible through the gitlab:zoekt rake tasks.

  • Adds enabled_namespaces_metrics to Search::Zoekt::MetricsService::METRICS.
  • Each run writes one zoekt.log line with meta.zoekt.enabled_namespaces_without_replicas set to Search::Zoekt::EnabledNamespace.without_replicas.count and metric: :enabled_namespaces_metrics.
  • MetricsUpdateCronWorker runs every minute, so this is one line per minute.
  • Adds the scope EnabledNamespace.without_replicas (NOT EXISTS on zoekt_replicas, using the index index_zoekt_replicas_on_enabled_namespace_id). It is extracted unchanged, with its model spec, from !257853 (merged), which uses it for rollout selection. Whichever MR merges second drops its copy on rebase.
  • Documents the new entry in the zoekt.log fields section of doc/administration/logs/_index.md.

To chart it in Kibana, filter json.metric: "enabled_namespaces_metrics" and plot the max of json.meta.zoekt.enabled_namespaces_without_replicas.

No feature flag: the change only logs and the query is read-only. Changelog: added, EE.

Query cost. One COUNT(*) anti-join per minute. On Database Lab it's a Parallel Hash Anti Join with index-only scans on zoekt_enabled_namespaces_pkey and index_zoekt_replicas_on_enabled_namespace_id. Warm: 52.9 ms, 53k buffer hits, no reads. Cold: 2.24 s, almost all of it disk I/O on the clone. Plan: https://console.postgres.ai/shared/dff3e998-ae05-48e2-b53c-01db79f1432c

On the Database Lab clone (2026-09-29), about 74,000 of about 190,000 enabled namespaces have no replica, with about 116,000 replicas in total.

What the count means, and why this design

The count is namespaces with no replica right now. That is mostly namespaces never rolled out. It also includes:

  • namespaces whose replica was later removed (rollout cleans up replicas without indices on each run, and eviction);
  • namespaces whose rollout is blocked by last_rollout_failed_at.

It does not include a namespace that has a replica but no indices yet.

Why not with_missing_indices: the goal is the rollout backlog, meaning namespaces never given a replica. without_replicas is the same set that rollout selection targets in !257853 (merged).

Why a log line and not a Prometheus gauge: it is the smallest change in this repo. A gauge would belong in the gitlab-exporter Zoekt probe (https://gitlab.com/gitlab-org/ruby/gems/gitlab-exporter/-/blob/master/lib/gitlab_exporter/database/zoekt.rb) and needs a release. It can follow later.

References

Screenshots or screen recordings

Not applicable. There are no UI changes.

How to set up and validate locally

  1. With exact code search indexing enabled and licensed, run in a Rails console: Search::Zoekt::MetricsUpdateCronWorker.new.perform('enabled_namespaces_metrics').
  2. Check log/zoekt.log for a line with "metric":"enabled_namespaces_metrics" and meta.zoekt.enabled_namespaces_without_replicas.

Tests: a model spec covers the scope. The service spec checks that a namespace with a replica is excluded and that two namespaces without replicas are counted.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Ravi Kumar

Merge request reports