Zoekt: Fix stuck overprovisioned watermark_level

What does this MR do and why?

This change adds a new feature to handle overprovisioned Zoekt search indices. It creates a new event system (ForceUpdateOverprovisionedIndexEvent) and a worker that identifies indices where the reserved_storage_bytes is outdated and needs to be recalculated. The worker updates these indices by recalculating their reserved_storage_bytes while skipping the used storage bytes calculation, which helps move overprovisioned ready indices back to a healthy state. The implementation includes:

  1. A new event class and worker for handling overprovisioned indices
  2. A new database scope to find indices with outdated reserved_storage_bytes information
  3. An enhancement to the existing update_storage_bytes_and_watermark_level! method to optionally skip updating used storage bytes
  4. Registration of the new worker in the Sidekiq queue configuration
  5. Comprehensive tests for the new functionality

This change improves the efficiency of the Zoekt search system by ensuring indices don't remain in an overprovisioned state when they don't need to be.

References

#528891 (closed)

Screenshots or screen recordings

Before After

How to set up and validate locally

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Query Report

https://console.postgres.ai/shared/d25696a8-01b3-4bf2-b8cc-9bf5539f52aa

SELECT
    "zoekt_indices".*
FROM
    "zoekt_indices"
WHERE
    "zoekt_indices"."watermark_level" = 10
    AND "zoekt_indices"."state" = 10
    AND (last_indexed_at < used_storage_bytes_updated_at)
LIMIT 1000
Edited by Ravi Kumar

Merge request reports

Loading