# Make Zoekt storage buffer factor dynamic ## Problem Zoekt's storage planner (`PlanningService`) uses a hardcoded `buffer_factor: 3` when computing how much disk to reserve per replica: ```ruby # ee/app/services/search/zoekt/planning_service.rb:248 def scaled_size(stats) stats.repository_size * buffer_factor # always 3× end ``` This `3×` reservation is stored in `zoekt_indices.reserved_storage_bytes` and directly determines how many namespaces can fit on a node (via `unclaimed_storage_bytes`). In practice, observed Zoekt disk usage on GitLab.com is closer to **`0.5×`** the source repository Git size. The `3×` reservation is intentionally conservative to ensure headroom during initial indexing — but once a namespace is indexed, the actual ratio is well-established and the reservation is significantly over-sized. This means nodes are under-utilized compared to their real capacity. ## Proposal Replace the hardcoded `buffer_factor: 3` with a **data-driven per-namespace ratio**. Once a namespace has been indexed, the actual expansion ratio is already tracked in `zoekt_enabled_namespaces.metadata['last_used_storage_bytes']`. Use this observed ratio (plus a safety margin) as the `buffer_factor` when re-planning, rather than the conservative `3×` default. ```ruby def dynamic_buffer_factor(enabled_namespace, safety_margin: 1.2) observed = enabled_namespace.metadata['last_used_storage_bytes'].to_f source = enabled_namespace.namespace.root_storage_statistics&.repository_size.to_f return 3.0 if source.zero? || observed.zero? # fallback for new/unindexed namespaces (observed / source) * safety_margin end ``` New namespaces (no observed data yet) fall back to `3×` until real usage data is available. Over time, reservations automatically converge toward actual usage patterns, improving node utilization without sacrificing headroom for growth. ## Current data model Key locations in the codebase: | Purpose | File | Line | |---|---|---| | Hardcoded `buffer_factor: 3` | `ee/app/services/search/zoekt/planning_service.rb` | 70 | | Formula: `repo_size × buffer_factor` | `ee/app/services/search/zoekt/planning_service.rb` | 248 | | Orphaned `BUFFER_FACTOR = 3` | `ee/app/services/search/zoekt/scheduling_service.rb` | 125 | | Caller (no buffer_factor passed) | `ee/app/services/search/zoekt/rollout_service.rb` | 39–43 | | `reserved_storage_bytes` stored (3× value) | `ee/app/services/search/zoekt/provisioning_service.rb` | 123 | | Actual usage: `used_storage_bytes` | `ee/app/models/search/zoekt/index.rb` | 163–166 | | Actual usage: `last_used_storage_bytes` | `ee/app/models/search/zoekt/enabled_namespace.rb` | 86–93 | ## Expected outcomes - Node utilization improves as the planner uses realistic reservations for already-indexed namespaces - New namespaces still get a safe `3×` initial reservation until observed data is available - Reservations automatically track actual usage patterns per namespace over time - No new settings or admin UI required ## Documentation Update `doc/integration/zoekt/_index.md` (the storage estimation section) to reflect the new behavior: - Explain that `3×` is the initial reservation for new namespaces - Explain that GitLab automatically adjusts the reservation downward as actual usage data becomes available - Update the examples to reflect that steady-state storage will be lower than the initial `3×` reservation