Zoekt: Fix stuck overprovisioned watermark_level
What does this MR do and why?
This change adds a new feature to handle overprovisioned Zoekt search indices. It creates a new event system (ForceUpdateOverprovisionedIndexEvent) and a worker that identifies indices where the reserved_storage_bytes is outdated and needs to be recalculated. The worker updates these indices by recalculating their reserved_storage_bytes while skipping the used storage bytes calculation, which helps move overprovisioned ready indices back to a healthy state. The implementation includes:
- A new event class and worker for handling
overprovisionedindices - A new database scope to find indices with outdated
reserved_storage_bytesinformation - An enhancement to the existing update_storage_bytes_and_watermark_level! method to optionally skip updating used storage bytes
- Registration of the new worker in the Sidekiq queue configuration
- Comprehensive tests for the new functionality
This change improves the efficiency of the Zoekt search system by ensuring indices don't remain in an overprovisioned state when they don't need to be.
References
Screenshots or screen recordings
| Before | After |
|---|---|
How to set up and validate locally
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Query Report
https://console.postgres.ai/shared/d25696a8-01b3-4bf2-b8cc-9bf5539f52aa
SELECT
"zoekt_indices".*
FROM
"zoekt_indices"
WHERE
"zoekt_indices"."watermark_level" = 10
AND "zoekt_indices"."state" = 10
AND (last_indexed_at < used_storage_bytes_updated_at)
LIMIT 1000