Monitoring for zoekt nodes and reservation capacity

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Work on this issue
• Close this issue

Background

We need a way to monitor zoekt node reservation capacity for monitoring and alerting.

Proposal

Add prometheus metrics to track:

• unclaimed storage bytes: search_zoekt_node_unclaimed_storage_bytes
• storage percent used: search_zoekt_node_storage_percent_used

question: is it worthwhile to also metrics to show WATERMARK_LIMIT_LOW, WATERMARK_LIMIT_HIGH, and WATERMARK_LIMIT_CRITICAL on graphs?

1. Add a new zoekt nodes graph to the global search grafana dashboard
   • See: https://gitlab.com/gitlab-com/runbooks/-/blob/master/dashboards/zoekt/main.dashboard.jsonnet
   • See: https://gitlab.com/gitlab-com/runbooks/-/blob/master/dashboards/search/zoekt.dashboard.jsonnet
2. update documentation with new metric: https://docs.gitlab.com/ee/administration/monitoring/prometheus/gitlab_metrics.html