2025-10-20: Zoekt search nodes are not enabled
Zoekt search nodes are not enabled (Severity 3 (Medium))
Problem: Zoekt search nodes went offline due to certificate verification errors, resulting in a site-wide reindexing process.
Impact: There was a significant drop in search requests and a spike in search latency between 14:30 and 16:00 UTC. All search nodes were offline for over 10 minutes, causing them to be marked as lost and triggering a full reindexing of all projects. This reindexing is nearly complete, with 99.6% of namespaces indexed as of 04:00 UTC. 118 namespaces remain without indices, but there are no longer any rollout-blocked namespaces. End users were not directly impacted since search requests used the advanced search fallback.
Causes: A recent helm chart change added custom certificate support, which led to TLS certificate verification errors ('certificate signed by unknown authority') in zoekt indexer logs. This caused the search nodes to report as offline.
Response strategy: We reverted the helm chart change that added custom certificate support, updated the GitLab Chart to use zoekt chart version 3.6.2, and deployed the fix. All zoekt nodes are now back online and functioning. We changed the 'Retry interval for failed namespaces' setting from 1 day to 10 minutes to accelerate progress, and full reindexing is nearly complete.
This ticket was created to track INC-5070, by incident.io