Add Prometheus metrics and documentation for Zoekt

What does this MR do and why?

Implements Prometheus instrumentation and documentation for Zoekt (exact code search) per-request and per-job metrics, as proposed in #591830 (closed) (part of epic gitlab-org#17919 (closed)).

This MR is intentionally scoped to in-process Rails metrics only. Database-derived Zoekt metrics already exist in gitlab-org/ruby/gems/gitlab-exporter under the search_zoekt_* prefix (see database/zoekt.rb) and are the right home for additional DB-state gauges. Three new exporter gauges are tracked in gitlab-org/ruby/gems/gitlab-exporter#120 (closed) as a follow-up.

Phase 1 — Audit findings

• Gitlab::Instrumentation::Zoekt.increment_request_count and .add_duration are called from ee/lib/gitlab/search/zoekt/client.rb on every HTTP request to a Zoekt node. The data is accumulated in SafeRequestStore and surfaced in log payloads (zoekt_calls, zoekt_duration_s) but is not emitted to Prometheus before this MR.
• Gitlab::Metrics::GlobalSearchSlis.record_apdex / .record_error_rate are already wired for Zoekt searches via SearchController and lib/api/search.rb. The ZOEKT_TARGET_S constant and 'zoekt' branch in duration_target are present. No new wiring needed.
• gitlab-exporter already emits 10 search_zoekt_* metrics covering node status, task processing queue, repository/index states, watermark levels, and per-node/per-index storage bytes. This MR does not add Rails-side duplicates of those metrics; additional DB-derived gauges are tracked separately in gitlab-exporter#120.
• A zoekt.log section already exists in doc/administration/logs/_index.md and is extended by this MR.

Phase 2 — Per-request Prometheus instrumentation (rack middleware + Sidekiq middleware)

Gitlab::Instrumentation::Zoekt remains a pure SafeRequestStore tally, mirroring Gitlab::Instrumentation::ElasticsearchTransport. Prometheus metrics are emitted via two separate paths:

Web / Grape requests — Gitlab::Metrics::ZoektRackMiddleware (mirrors ElasticsearchRackMiddleware) reads query_time and get_request_count at end-of-request and emits:

• http_zoekt_requests_total (Counter, labels: controller, action, endpoint_id) — Amount of calls to Zoekt servers during web transactions.
• http_zoekt_requests_duration_seconds (Histogram, labels: controller, action, endpoint_id) — Query time for Zoekt servers during web transactions.

Sidekiq jobs — Gitlab::SidekiqMiddleware::ServerMetrics reads zoekt_calls and zoekt_duration_s from the instrumentation payload (already populated by Gitlab::InstrumentationHelper#instrument_zoekt) and emits:

• sidekiq_zoekt_requests_total (Counter) — Zoekt requests during a Sidekiq job execution.
• sidekiq_zoekt_requests_duration_seconds (Histogram) — Duration in seconds that a Sidekiq job spent in requests to a Zoekt server.

All four additions mirror the existing Elasticsearch pattern 1:1.

All existing SafeRequestStore logic and log payloads (zoekt_calls, zoekt_duration_s) are preserved.

Phase 3 — SLI verification

Confirmed that record_apdex / record_error_rate are already called for Zoekt searches in SearchController and lib/api/search.rb. The ZOEKT_TARGET_S = 15.52 constant and 'zoekt' branch in duration_target are present in lib/gitlab/metrics/global_search_slis.rb. No code changes needed.

Phase 4 — Documentation

• Added a Zoekt metrics section to doc/administration/monitoring/prometheus/gitlab_metrics.md with a single per-request-and-per-job table covering all four new metrics (http_zoekt_requests_* and sidekiq_zoekt_requests_*), plus the existing global search SLI metrics and Zoekt task SLI metrics. A note points readers to gitlab-exporter for database-derived metrics.
• Added a zoekt.log fields subsection to doc/administration/logs/_index.md documenting log payload fields for node metrics entries, indices metrics entries, and per-request fields.

Backward compatibility

Log emissions are preserved unchanged. Prometheus metrics are additive. No existing metric names are reused or shadowed.

Follow-ups (out of scope for this MR)

• gitlab-exporter#120 tracks adding DB-derived gauges in the exporter: enabled namespaces per node, full task-state breakdown per node (one GROUP BY query that supersedes the existing search_zoekt_task_processing_queue_size), and the stale-used-storage-bytes index count.
• Alert rules for these metrics should be added in gitlab-com/runbooks separately.

References

• Issue: #591830 (closed)
• Parent epic: gitlab-org#17919 (closed)
• Exporter follow-up: gitlab-org/ruby/gems/gitlab-exporter#120 (closed)

Screenshots or screen recordings

How to set up and validate locally

1. Enable Zoekt search on a self-managed instance.
2. Perform a code search using Zoekt.
3. Check /-/metrics for http_zoekt_requests_total and http_zoekt_requests_duration_seconds.
4. Run a Sidekiq job that performs Zoekt searches and check sidekiq_zoekt_requests_total and sidekiq_zoekt_requests_duration_seconds.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.