Reserve Zoekt indices above the critical watermark

What does this MR do and why?

DYNAMIC_BUFFER_SAFETY_MARGIN was inconsistent with STORAGE_CRITICAL_WATERMARK, so Zoekt indices were provisioned already over the eviction threshold.

PlanningService reserves repository_size × buffer_factor, and buffer_factor already includes the margin. Once indexing completes, used_storage_bytes ≈ repository_size × ratio, so:

used / reserved = 1 / DYNAMIC_BUFFER_SAFETY_MARGIN

At 1.2 that is 0.8333, above STORAGE_CRITICAL_WATERMARK of 0.80. Even with a perfectly accurate ratio, a new index became critical_watermark_exceeded the moment indexing finished — any margin at or below 1 / 0.80 = 1.25 does this. Index.should_be_pending_eviction selects exactly that level, so SchedulingService marks the index pending_eviction and IndexToEvictEventWorker evicts it and deletes its replica. Because that worker evicts by replica, healthy siblings go too.

An index survived only by winning a race: refresh_reserved_storage_bytes retargets the reservation to used / STORAGE_IDEAL_PERCENT_USED, but that growth goes through claim_reserved_storage_bytes_from_node, which caps at [ideal, unclaimed + current].min and silently declines when the node is full.

The margin is now derived from STORAGE_IDEAL_PERCENT_USED rather than hardcoded, so the two constants cannot drift apart again:

DYNAMIC_BUFFER_SAFETY_MARGIN = (1 / STORAGE_IDEAL_PERCENT_USED) * 1.5 # => 2.5

It is declared after the STORAGE_* constants rather than in alphabetical position because Ruby resolves constants top to bottom, and the expression references one of them.

An earlier revision of this MR asserted the two bounds arithmetically. Those examples are gone: both divided by the margin and compared the result against STORAGE_IDEAL_PERCENT_USED, the same constant the margin is derived from, so that constant cancelled out on both sides — one reduced to the identity 1.5 >= 1, which cannot fail for any value of that constant.

They're replaced by one example that provisions an index through the real code path: it seeds a ready index so Index.global_buffer_factor returns an observed ratio, reserves source_bytes × global_buffer_factor for a new index, fills that index to the size the factor predicted, and asserts the watermark level it lands on:

expect(new_index.appropriate_watermark_level).to be_in(%i[overprovisioned healthy])

This fails red against the old 1.2 (expected :critical_watermark_exceeded.in?([:overprovisioned, :healthy])) and passes green against 2.5. It's also sensitive to the range boundaries in appropriate_watermark_level, which the arithmetic examples never touched.

It's still insensitive to STORAGE_IDEAL_PERCENT_USED itself, since the margin is defined relative to that constant and the watermark bands key off it too, so they co-vary by design. That's consistent with GitLab's RSpec guidance: test the behaviour that depends on a constant, not the constant's value.

Why 2.5

Margin	used/reserved	Underestimate tolerated
1.2 (before)	0.8333	under 0.96x — fails even at a perfect estimate
1.25	0.8000	under 1.00x — critical at a perfect estimate
1.6667 (1/0.6)	0.6000	under 1.33x
2.5 (this MR)	0.4000	under 2.00x

The bound is strict because appropriate_watermark_level uses ... (end-exclusive) ranges with critical_watermark_exceeded in the else, so used/reserved == 0.80 exactly still counts as critical.

The instance ratio is a size-weighted mean, and a namespace observed in production ran at 0.5752 against an instance ratio of 0.3290 — 1.75x. Margin 2.5 covers that; 1.6667 does not.

The extra reservation is front-loaded, not wasted: once an index reaches ready, refresh_reserved_storage_bytes retargets down to used / 0.6 and releases the excess. overprovisioned indices are never evicted.

Production evidence

Index 2991031, two samples 14 minutes apart, both critical_watermark_exceeded:

	17:38:00	17:52:05
reserved_storage_bytes	161,184,787,076	161,348,653,812
used_storage_bytes	169,939,386,615	235,088,873,496
used/reserved	1.0543	1.4570

used grew 65 GB while the reservation moved 164 MB — the node had no unclaimed space, so growth was starved. The index was still indexing at the second sample, so its ratio is a lower bound.

Risk

Reservations grow 2.5 / 1.2 = 1.67x. Consequences:

1. Nodes admit less, so some namespaces will fail at plan time with node_unavailable rather than being provisioned and later evicted. Better failure mode, but a behaviour change.
2. MAX_INDICES_PER_REPLICA is 10, and a namespace now needs proportionally more indices, so one previously needing 6 or more could hit index_limit_exceeded. Worth checking the current index-per-replica distribution.
3. find_best_node requires one node with unclaimed >= repository_size × ratio × margin per project, so a very large repository can fail placement where it previously fit.

Out of scope, tracked separately: compute_global_buffer_factor divides ready-index bytes by whole-namespace repository_size, biasing the instance ratio low; IndexToEvictEventWorker evicting healthy siblings by replica; and PlanningService concentrating most of a namespace into one index. None of them fix this bug, because used/reserved converges to 1/margin regardless of ratio accuracy.

References

No linked issue yet — raised from analysis of a repeated production eviction.

Screenshots or screen recordings

Not applicable, backend-only change with no UI surface.

How to set up and validate locally

1. Temporarily set DYNAMIC_BUFFER_SAFETY_MARGIN = 1.2 in ee/app/models/search/zoekt/index.rb, run bundle exec rspec ee/spec/models/search/zoekt/index_spec.rb -e 'provisioning from the dynamic buffer factor', confirm it fails with :critical_watermark_exceeded, then restore it.
2. Run bundle exec rspec ee/spec/models/search/zoekt/index_spec.rb.
3. After a rollout, check Search::Zoekt::Index.pluck(:reserved_storage_bytes, :used_storage_bytes, :watermark_level) — nothing should land in critical_watermark_exceeded once indexing completes.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.