Monitoring for zoekt indices

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Work on this issue
• Close this issue

Background

We need a way to check zoekt indices watermark level states for zoekt indices for monitoring and alerting.

Proposal

Add prometheus metrics to track:

• guage to count indices in each state: search_zoekt_indices_states
• gauge to count indices in each watermark_level: search_zoekt_indices_watermark_levels
• reserved storage bytes for each index: search_zoekt_indices_watermark_levels
• reserved_storage_bytes for each index: search_zoekt_indices_reserved_storage_bytes
• used_storage_bytes for each index: search_zoekt_indices_used_storage_bytes

1. Add a new zoekt indices graph to the global search grafana dashboard
   • See: https://gitlab.com/gitlab-com/runbooks/-/blob/master/dashboards/zoekt/main.dashboard.jsonnet
   • See: https://gitlab.com/gitlab-com/runbooks/-/blob/master/dashboards/search/zoekt.dashboard.jsonnet
2. update documentation with new metrics: https://docs.gitlab.com/ee/administration/monitoring/prometheus/gitlab_metrics.html