RolloutWorker error "No node can accommodate project ... with size ..."

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Summary

RolloutWorker thinks that there is not enough space and refuses to continue. After enabling Zoekt the log shows:

Configuring Zoekt manually fixes that issue.

Steps to reproduce

Install Zoekt with Docker-compose, Enable Zoekt indexing in the settings, check the zoekt.log

What is the current bug behavior?

Search::Zoekt::RolloutWorker thinks there is no space available to index repositories

What is the expected correct behavior?

Relevant logs and/or screenshots

{
  "severity": "INFO",
  "time": "2025-03-28T15:35:16.006Z",
  "meta.caller_id": "Search::Zoekt::RolloutWorker",
  "correlation_id": "abbe16581b3c35c71b58fbd774b2afaf",
  "meta.root_caller_id": "Cronjob",
  "meta.feature_category": "global_search",
  "meta.client_id": "ip/",
  "message": "{\"total_required_storage_bytes\":0,\"namespaces\":[],\"nodes\":[{\"id\":2,\"unclaimed_storage_bytes_before\":17786686169,\"unclaimed_storage_bytes_after\":17786621363,\"claimed_storage_bytes\":64806,\"namespace_ids\":[2],\"indices\":[{\"node_id\":2,\"namespace_id\":2,\"projects\":{\"project_namespace_id_from\":4,\"project_namespace_id_to\":null},\"required_storage_bytes\":64806,\"max_storage_bytes\":17786686169}]}],\"failures\":[{\"namespace_id\":2,\"enabled_namespace_id\":4,\"namespace_required_storage_bytes\":64806,\"replicas\":[{\"indices\":[{\"node_id\":2,\"projects\":{\"project_namespace_id_from\":4,\"project_namespace_id_to\":null},\"required_storage_bytes\":64806,\"max_storage_bytes\":17786686169}]}],\"errors\":[{\"namespace_id\":2,\"replica_idx\":0,\"type\":\"node_unavailable\",\"details\":\"No node can accommodate project 2 with size 108153\"},{\"namespace_id\":2,\"replica_idx\":0,\"type\":\"node_unavailable\",\"details\":\"No node can accommodate project 3 with size 446946\"}]}]}"
}

zoekt.log

gitlab:zoekt:info shows:

Exact Code Search
GitLab version:                                 17.10.0
Enable indexing:                                yes
Enable searching:                               yes
Pause indexing:                                 no
Index root namespaces automatically:            yes
Delete offline nodes after 12 hours:            no
Indexing CPU to tasks multiplier:               1.0
Batch size of namespaces for initial indexing:  32

Nodes
Node count:                    2 (online: 1, offline: 1)
Last seen at:                  2025-03-28 15:33:32 UTC (less than a minute ago)
Storage reserved / usable:     0 B / 16.6 GiB (0.0%)
Storage indexed / reserved:    1.39 MiB / 0 B (0%)
Storage used / total:          21.7 GiB / 57.8 GiB (37.54%)
Online node watermark levels:  1
  - normal: 1

Indexing status
EnabledNamespace count:            1 (without indices: 1, with search disabled: 0)
Replicas count:                    (none)
Indices count:                     (none)
Indices watermark levels:          (none)
Repositories count:                (none)
Tasks count:                       (none)
Tasks pending/processing by type:  (none)

Feature Flags (Non-Default Values)
Feature flags:  none

Feature Flags (Default Values)
- zoekt_create_multiple_indices:           disabled
- zoekt_critical_watermark_stop_indexing:  disabled
- zoekt_cross_namespace_search:            disabled
- zoekt_index_circuit_breaker:             disabled
- zoekt_index_pending_delete_repos:        enabled
- zoekt_multimatch_frontend:               disabled
- zoekt_node_assignment:                   disabled
- zoekt_node_backoffs:                     disabled
- zoekt_optimized_performance_indexing:    disabled
- zoekt_reduced_pull_frequency:            enabled
- zoekt_rollout_worker:                    enabled
- zoekt_search_api:                        enabled
- zoekt_search_proxy:                      disabled

Possible fixes